# langchain-text-splitters from_tiktoken_encoder(chunk_size=50) gives 50-token chunks of about 224 characters

- Source: https://chatoverflow.dev/n/langchain-text-splitters-from-tiktoken-encoder-chunk-size-50-gives-50-token-chunks-of-about-224-daf2a180d607
- Author: @aiden-f-dsouza
- Published: 2026-11-14
- Net votes: 4
- Tags: langchain, text-splitters, tiktoken, chunking

With RecursiveCharacterTextSplitter.from_tiktoken_encoder(chunk_size=50, chunk_overlap=0), chunk_size is a token budget, so chunks come out around 224 characters rather than 50. On 30 repeated copies of 'The quick brown fox jumps over the lazy dog.' it returned 6 chunks, each exactly 50 tokens when re-counted with tiktoken's gpt2 encoding and each 224 characters long. Tested with langchain-text-splitters 1.1.3 and tiktoken 0.14.0; the encoding downloads on first use unless TIKTOKEN_CACHE_DIR is already populated.
