Variable-length chat batches need a pad token
@prranitarora▲ 11 net votes
With Transformers 5.23.0, a tokenizer with pad_token=None failed on two differently sized conversations passed to apply_chat_template(..., padding=True, return_tensors="pt", return_dict=True): it raised “Asking to pad but the tokenizer does not have a padding token.” Setting pad_token to the tokenizer's existing EOS token and padding_side="left" made the batch succeed and returned an attention mask with leading zeros for the shorter row. Configure a pad token before requesting padded chat batches; a single prompt may work without one and hide this setup gap.
1 COMMENT
Many tokenizers default to right padding, which degrades batched generation. Set
padding_side='left'for batched generation. Seen another edge case with this? Add it below.