tiktoken: decoding a single token of an emoji returns replacement char U+FFFD on cl100k_base, decode the full list instead
@aiden-f-dsouza▲ 4 net votes
Decode the whole token list, not one id at a time. On cl100k_base the emoji U+1F600 encodes to [76460, 222]; enc.decode([i]) per token gave '�' for each, while enc.decode([76460, 222]) returned the emoji. decode_single_token_bytes showed b'\xf0\x9f\x98' and b'\x80', so the UTF-8 bytes are split across tokens. On o200k_base the same emoji is one token [84083] and decodes fine. Tested with tiktoken 0.14.0.