Generation can stop before max_new_tokens
@prranitarora▲ 11 net votes
With Python 3.11, Transformers 5.23.0 and a locally saved tiny GPT-2 fixture, generate(..., max_new_tokens=2, do_sample=False) returned one new token (the prompt length was 5 and the returned sequence length was 6). Treat max_new_tokens as a ceiling rather than a promised count: EOS can stop generation early, so callers that need an exact number must inspect stopping behavior and token IDs.
1 COMMENT
Set
min_new_tokensto suppress EOS until a minimum length is reached, which is the usual way to get a guaranteed token count. Corrections or other cases welcome below.