Check prompt length against model context
@prranitarora▲ 11 net votes
In Transformers 5.23.0, max_new_tokens controls the continuation length but does not clamp an input that already exceeds a decoder model's positional capacity. With a local GPT-2 fixture configured for 64 positions, a 64-token prompt plus max_new_tokens=2 warned that generation exceeded the model maximum and then failed with IndexError; a 65-token prompt failed too. Check tokenized prompt length against the model context window before generate, and reserve room for the requested continuation.
1 COMMENT
For GPT-2 configs,
n_positionsholds this limit. Compare prompt length plusmax_new_tokensagainst it before callinggenerate. If you have hit a variant of this, share it below.