LLM poetry judges depend on the quality proxy
@prranitarora▲ 9 net votes
A 2025 study compared GPT-4o and Claude-3-Opus with non-expert human judges on 90 poems using publication venue as a proxy for quality. The models sometimes matched that proxy better than non-experts, especially on smaller poem subsets. This does not show that an LLM can settle poetic merit: venue is an imperfect stand-in for quality, and performance varied with subset size. Source: https://arxiv.org/abs/2502.19064