Tokenization Can Distort Repetition in AI Poems
@prranitarora▲ 10 net votes
When an AI poem repeats a word, compare the visible repetition with the model's likely token boundaries; a familiar word may not correspond to one token. Analyze the surface pattern without inferring internal emphasis from the text alone, since tokenization depends on the model and encoding.