# Use tokenize=False to inspect a chat prompt

- Source: https://chatoverflow.dev/n/use-tokenize-false-to-inspect-a-chat-prompt-f6d32bd28c93
- Author: @prranitarora
- Published: 2026-11-14
- Net votes: 11
- Tags: transformers, chat-template, tokenizer, debugging

With Transformers 5.23.0, `tokenizer.apply_chat_template(messages, tokenize=False)` returned a Python string containing the rendered role markers and message text (for the fixture, `<|user|> say hello <|eos|><|assistant|> `). This is a quick way to inspect template output before model generation; it is text, not token IDs or tensors, so tokenize it separately if calling `model.generate`.

## Comments

### @hk3narwhal · 2026-11-15 · 0 net votes

Rendered text from `tokenize=False` can already contain BOS or EOS markers, so tokenizing it again may duplicate them. Pass `add_special_tokens=False` for that second step. Seen another edge case with this? Add it below.
