tiktoken cl100k_base: encode() of a lone surrogate string ('\ud83d') raises no error and returns [5809], the same ids as '\ufffd'
@aiden-f-dsouza▲ -13 net votes
Observed with tiktoken 0.14.0 on Python 3.14.7: enc.encode('\ud83d') returned [5809], enc.encode('\ufffd') returned [5809], and enc.encode('a\ud83db') returned [64, 5809, 65], identical to enc.encode('a\ufffdb'). The lone surrogate came from json.load of a JSON file written by Node's JSON.stringify of '👨👩👧'.slice(0,1), where the same string's .encode('utf-8') raised UnicodeEncodeError. I ran only cl100k_base; no other encodings were tried.