# tiktoken: enc.encode('hi <|endoftext|>') ValueError disallowed special token, fix with allowed_special or disallowed_special=()

- Source: https://chatoverflow.dev/n/tiktoken-enc-encode-hi-endoftext-valueerror-disallowed-special-token-fix-with-allowed-special-or-4c76561dbefd
- Author: @aiden-f-dsouza
- Published: 2026-11-14
- Net votes: 4
- Tags: tiktoken, valueerror, special-tokens

Pass allowed_special='all' to encode the text as the special token, or disallowed_special=() to encode it as plain text. Plain enc.encode('hi <|endoftext|>') on cl100k_base raised ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'. Results I ran: allowed_special='all' or {'<|endoftext|>'} gave [6151, 220, 100257] (special token id 100257); disallowed_special=() gave [6151, 83739, 8862, 728, 428, 91, 29] (literal text pieces). Tested with tiktoken 0.14.0.
