Prior over Evidence: Stereotype-Driven Diagnosis in LLM-Based L2 Pronunciation Feedback

2026-06-13

AI Summary

This study investigates whether large language models (LLMs) rely on input speech evidence or are driven by pretraining-induced priors when providing pronunciation feedback for second-language (L2) English learners. Through systematic experiments, the authors evaluate three audio-aware LLMs across multiple evidence conditions—text, acoustic features, and raw audio—using 1,800 utterances from the L2-Arctic corpus. The work reveals a previously undocumented phenomenon: LLMs exhibit high internal consistency but low diagnostic accuracy, consistently attributing errors to a fixed set of phonemes regardless of speakers’ native language backgrounds or variations in input evidence. Notably, scoring accuracy improves significantly only when textualized acoustic features directly relevant to the target phonetic dimension are provided. These findings suggest that LLMs are better suited as natural language interfaces for external pronunciation analyses rather than as standalone diagnostic engines.

Abstract

Large language models are increasingly deployed for written pronunciation feedback in second-language (L2) English learning, under the assumption that their diagnoses are grounded in the supplied speech evidence rather than in priors from pretraining. This assumption is tested on 1,800 L2-Arctic utterances spanning six L1 backgrounds, three audio-capable LLMs, four pronunciation dimensions, and five evidence conditions ranging from a text-only baseline to numeric acoustic features and raw audio. Each (utterance x model x condition x dimension) cell is scored on three metrics: Rating Accuracy (RA) against gold labels, Evidence Coherence (EC) assessing internal consistency without ground truth, and Grounded Correctness (GC) evaluated against gold evidence. Results show three findings across models. First, rating accuracy and grounded reasoning decouple: 39.6% of judged cells contain internally coherent reasoning that supports a wrong rating, against only 15.8% where the reasoning supports a correct rating. Second, phoneme-level feedback converges to a fixed inventory of L2-English difficulty phones that recurs across all six L1 backgrounds and all evidence conditions. Third, acoustic evidence improves the rating only when the supplied feature directly probes the target dimension: textualised F0 range raises pitch-variation grounding from (0.18-0.19) to (0.45-0.62) across all three models, while stress and phoneme correctness, which require target-to-realisation alignment, remain ungrounded. The same audio waveform without textualised F0 values does not reproduce this improvement. These findings indicate that current general-purpose LLMs are more reliable as verbalisers of externally computed pronunciation evidence than as standalone diagnostic engines.

Problem

• stereotype
• pronunciation feedback
• large language models
• evidence grounding
• L2 learning

Innovation

• stereotype-driven diagnosis
• evidence grounding
• LLM-based pronunciation feedback
• acoustic feature textualization
• L2 pronunciation assessment