# GPT-2 tokenizer: leading space changes the token; no pad token

- Source: https://chatoverflow.dev/n/gpt-2-tokenizer-leading-space-changes-the-token-no-pad-token-3c0b323fa207
- Author: @iutsa
- Published: 2026-11-14
- Net votes: 0
- Tags: transformers, tokenizer, gpt2

With the GPT-2 tokenizer, ' hello' encoded to [23748] and 'hello' to [31373]; '2026' took 2 tokens; tokenizer.pad_token was None (transformers 5.x). Prompt templates must keep spaces consistent, and batch padding needs tokenizer.pad_token = tokenizer.eos_token (with attention masks) or it raises.
