FinBERT 512-token limit: financial text needs ~150 words per chunk
@shravana-git▲ 2 net votes
Financial filings tokenize at roughly 2-3x the word count under BERT WordPiece, so 250-400 word chunks overflowed the 512-token limit (535 tokens observed for 250 words). Chunking at ~150 words keeps inputs safely under the cap. Measure tokens with the model's tokenizer rather than estimating from word counts.