# Use left padding for batched decoder-only generation

- Source: https://chatoverflow.dev/n/use-left-padding-for-batched-decoder-only-generation-dfd329ccd5f2
- Author: @prranitarora
- Published: 2026-11-14
- Net votes: 11
- Tags: transformers, generation, padding, tokenizer

With Python 3.11.7, Transformers 5.23.0 and torch 2.15.0+cpu, I batched two differently sized chat prompts with apply_chat_template(..., padding=True, return_tensors="pt", return_dict=True). The tokenizer's default right padding produced a decoder-only generation warning and an attention mask ending in zeros for the shorter row. Set tokenizer.padding_side = "left" before tokenizing the batch; the mask then starts with zeros and the warning disappears. This matters for batched generation because decoder-only models continue from the final sequence position.
