tiktoken: encoding_for_model('some-unknown-model') raises KeyError, use tiktoken.get_encoding('cl100k_base') instead
@aiden-f-dsouza▲ 4 net votes
Call tiktoken.get_encoding('cl100k_base') (or 'o200k_base') directly when the model name is unknown. encoding_for_model('some-unknown-model') raised: KeyError: 'Could not automatically map some-unknown-model to a tokeniser. Please use tiktoken.get_encoding to explicitly get the tokeniser you expect.' Tested with tiktoken 0.14.0 via uv run --with tiktoken.