LLM-Based Multi-Reference Evaluation for Efficient and Robust Assessment of Phrase Break Annotations

2026-06-19

AI Summary

Existing evaluation methods for prosodic phrase segmentation struggle to balance segmentation diversity and assessment efficiency: single-reference approaches overlook the inherent variability of valid segmentations, while human evaluation is costly and lacks scalability. This work introduces large language models into the automatic evaluation of spoken prosodic phrasing, proposing a few-shot, multi-reference evaluation framework that generates diverse yet plausible segmentations from a small set of examples and scores them using a multi-reference alignment strategy. Evaluated on a Korean test set comprising 1,356 annotated utterances, the proposed method significantly outperforms single-reference baselines in both correlation with human judgments and scoring consistency, achieving a favorable trade-off between scalability and fidelity to human-like assessment.

Abstract

Reliable evaluation of phrase break annotations is crucial, as subtle variations in prosodic boundaries directly affect the clarity and naturalness of speech. However, existing approaches exhibit major limitations: single-reference evaluation assumes a unique gold phrasing for an utterance despite multiple valid phrasings, while human judgment, though flexible, is labor-intensive and unscalable. To address these, we propose LLM-based Multi-Reference Evaluation (LMRE) for phrase break annotations that models the one-to-many nature of prosodic phrasing and generates multiple valid phrasings from minimal demonstrations. On a Korean testbed of 1,356 annotations covering five strategies, LMRE shows stronger alignment with human judgment than single-reference evaluation in both acceptance behavior and score correlation. Our findings demonstrate that LMRE effectively achieves both scalability and multi-reference support, highlighting the potential of LLMs for evaluation in the speech domain.

Problem

• phrase break annotation
• multi-reference evaluation
• prosodic phrasing
• speech evaluation
• LLM-based assessment

Innovation

• LLM-based evaluation
• multi-reference assessment
• phrase break annotation
• prosodic phrasing
• speech evaluation