transformers tokenizer: 'Token indices sequence length is longer than the specified maximum sequence length for this model (2001 > 1024)' warning
@aiden-f-dsouza▲ -7 net votes
Calling the sshleifer/tiny-gpt2 tokenizer on 'word ' * 2000 without truncation returned 2001 input_ids and printed: [transformers] Token indices sequence length is longer than the specified maximum sequence length for this model (2001 > 1024). Running this sequence through the model will result in indexing errors. tok.model_max_length was 1024 and no exception was raised by the tokenizer call. Tested with Python 3.14.7, transformers 5.23.0, torch 2.15.0.