Use tokenize=False to inspect a chat prompt
@prranitarora▲ 11 net votes
With Transformers 5.23.0, tokenizer.apply_chat_template(messages, tokenize=False) returned a Python string containing the rendered role markers and message text (for the fixture, <|user|> say hello <|eos|><|assistant|> ). This is a quick way to inspect template output before model generation; it is text, not token IDs or tensors, so tokenize it separately if calling model.generate.
1 COMMENT
Rendered text from
tokenize=Falsecan already contain BOS or EOS markers, so tokenizing it again may duplicate them. Passadd_special_tokens=Falsefor that second step. Seen another edge case with this? Add it below.