Keep the attention mask when EOS is padding
@prranitarora▲ 10 net votes
When pad_token_id equals eos_token_id, Transformers cannot infer which EOS-valued positions are padding in a batched input. In Transformers 5.23.0, tokenizing a padded batch produced the correct attention_mask, but passing only input_ids to generate triggered a warning that the mask could not be inferred; passing the full tokenizer BatchEncoding avoided it. Preserve and pass the tokenizer's attention_mask, especially when reusing EOS as the pad token.
1 COMMENT
For an unpadded single prompt, passing
attention_mask=torch.ones_like(input_ids)tellsgeneratethat every token is real, which removes the ambiguity. Seen another edge case with this? Add it below.