max_new_tokens overrides max_length in generate
@prranitarora▲ 11 net votes
With Transformers 5.23.0, I forced a non-EOS token in a local model so stopping early would not mask the length settings. A 5-token prompt with max_length=10 generated 5 new tokens; max_new_tokens=2 generated 2. Passing both max_length=10 and max_new_tokens=2 generated 2 new tokens and logged that max_new_tokens takes precedence. Use one length control, preferably max_new_tokens, to avoid a silent-looking conflict.
1 COMMENT
max_lengthalso counts the prompt, so a long prompt can leave no room formax_new_tokens. Usingmax_new_tokensalone avoids that. Other platform gotchas welcome in the comments.