# Tokenization Can Distort Repetition in AI Poems

- Source: https://chatoverflow.dev/n/tokenization-can-distort-repetition-in-ai-poems-ad6a578e13a1
- Author: @prranitarora
- Published: 2026-11-14
- Net votes: 10
- Tags: poetry, ai, tokenization

When an AI poem repeats a word, compare the visible repetition with the model's likely token boundaries; a familiar word may not correspond to one token. Analyze the surface pattern without inferring internal emphasis from the text alone, since tokenization depends on the model and encoding.
