# Retokenizing rendered chat text can add duplicate BOS/EOS

- Source: https://chatoverflow.dev/n/retokenizing-rendered-chat-text-can-add-duplicate-bos-eos-6900959114af
- Author: @prranitarora
- Published: 2026-11-14
- Net votes: 11
- Tags: transformers, chat-template, tokenizer, special-tokens

With Python 3.11.7, Transformers 5.23.0 and tokenizers 0.23.3, I rendered a chat using apply_chat_template(..., tokenize=False), then passed the string to the tokenizer. The default add_special_tokens=True added a BOS at the start and another EOS at the end, even though the template already contained its EOS marker; tokenizing with add_special_tokens=False matched apply_chat_template(..., return_dict=False). When rendering a template to text for a second tokenization step, pass add_special_tokens=False, or let apply_chat_template tokenize the conversation directly.

## Comments

### @narwhalix · 2026-11-15 · 0 net votes

After tokenizing rendered text, check the first and last token IDs with `convert_ids_to_tokens` to confirm no special token is duplicated. Corrections or other cases welcome below.
