Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?

2026-06-29

AI Summary

Existing automatic evaluation methods struggle to effectively assess the quality of modern Chinese poetry comprehension, while human evaluation remains costly and difficult to scale. To address this challenge, this work proposes Poller, a novel approach that, for the first time, endows large language models with the persona of a poet to guide multidimensional, author-perspective automated evaluation, thereby simulating human judgment processes. By constructing a role-playing–based evaluation framework, Poller significantly reduces assessment errors by 94.55% and 89.53% on key dimensions such as rhetorical devices and defamiliarization, respectively, markedly narrowing the gap with human evaluations. This method establishes a new, efficient, and reliable paradigm for automated evaluation in poetry comprehension tasks.

Abstract

Traditional automatic evaluation methods have been shown to be unsuitable for modern Chinese poetry because of the distinct nature of this literary genre. Human evaluation remains reliable, but is expensive and not applicable to large-scale data. In this paper, we propose Poller (Poetry LLM Evaluator), a novel method leveraging large language models (LLMs) to evaluate the poetry understanding task. Specifically, our method requires LLMs to play the role of a poem's author with detailed information, thereby emulating human evaluation and judgment by adopting the poet's perspective. We conducted comprehensive experiments on multiple LLMs, evaluating the interpretations of poems across eight specialized dimensions. Experimental results demonstrate that our method effectively reduces the evaluation error between LLMs and humans. Especially for specific dimension evaluation, Poller-based LLMs achieve a 94.55% and 89.53% error reduction for rhetorical techniques and defamiliarization, respectively, compared to baseline methods. These performances are unattainable by conventional LLM evaluation methods. Experimental results from multiple LLMs across various dimensions validate the efficacy of our method. This work bridges the gap between automated efficiency and human expertise, establishing a foundation for automated evaluation in poetry-related tasks.

Problem

• poetry understanding
• automatic evaluation
• large language models
• human evaluation
• modern Chinese poetry

Innovation

• LLM-based evaluation
• poetry understanding
• author perspective simulation
• automated literary assessment
• defamiliarization