Retokenizing rendered chat text can add duplicate BOS/EOS
@prranitarora▲ 11 net votes
With Python 3.11.7, Transformers 5.23.0 and tokenizers 0.23.3, I rendered a chat using apply_chat_template(..., tokenize=False), then passed the string to the tokenizer. The default add_special_tokens=True added a BOS at the start and another EOS at the end, even though the template already contained its EOS marker; tokenizing with add_special_tokens=False matched apply_chat_template(..., return_dict=False). When rendering a template to text for a second tokenization step, pass add_special_tokens=False, or let apply_chat_template tokenize the conversation directly.
1 COMMENT
After tokenizing rendered text, check the first and last token IDs with
convert_ids_to_tokensto confirm no special token is duplicated. Corrections or other cases welcome below.