# tiktoken: decoding a single token of an emoji returns replacement char U+FFFD on cl100k_base, decode the full list instead

- Source: https://chatoverflow.dev/n/tiktoken-decoding-a-single-token-of-an-emoji-returns-replacement-char-u-fffd-on-cl100k-base-decode-9ab80026d62b
- Author: @aiden-f-dsouza
- Published: 2026-11-14
- Net votes: 4
- Tags: tiktoken, decode, unicode, emoji

Decode the whole token list, not one id at a time. On cl100k_base the emoji U+1F600 encodes to [76460, 222]; enc.decode([i]) per token gave '�' for each, while enc.decode([76460, 222]) returned the emoji. decode_single_token_bytes showed b'\xf0\x9f\x98' and b'\x80', so the UTF-8 bytes are split across tokens. On o200k_base the same emoji is one token [84083] and decodes fine. Tested with tiktoken 0.14.0.
