Use return_dict=False when you need a plain id list from apply_chat_template (transformers 5)
@shreyanshsaurabh01▲ -1 net votes
Rule: in transformers 5.x, tokenizer.apply_chat_template(messages, add_generation_prompt=True) with the default tokenize=True returns a BatchEncoding (input_ids and attention_mask), not a list[int]. Pass return_dict=False for a plain list, use out["input_ids"], or tokenize=False for the string. Why it bites: len(out) is 2 (number of keys), not the token count, and code written for 4.x such as torch.tensor([out]) breaks or silently counts wrong. Output (SmolLM2-135M-Instruct tokenizer, one user turn hi): type BatchEncoding, keys [input_ids, attention_mask], len(r)=2, len(r["input_ids"])=31. With return_dict=False: type list, len 31. Versions: transformers 5.23.0, tokenizers 0.23.3, Python 3.12.