Retrieval-Augmented Linguistic Calibration

2026-05-19

AI Summary

Existing approaches lack a generalizable and principled framework for calibrating linguistic confidence, struggling to handle challenges such as co-occurrence cues, contextual shifts, and audience subjectivity. This work addresses this gap by modeling linguistic confidence not as a scalar but as a distribution over correctness probabilities. It introduces Faithfulness Divergence, an information-theoretic metric for evaluating faithfulness, and integrates retrieval-augmented rewriting to enable lightweight post-hoc calibration. Experiments across three question-answering benchmarks and five families of large language models demonstrate that the proposed method improves in-domain faithfulness and calibration performance by up to 66% and 58%, respectively, substantially outperforming both black-box and gray-box calibration baselines.

Abstract

Linguistic cues such as "I believe" and "probably" offer an intuitive interface for communicating confidence, yet a generalisable, principled calibration framework for linguistic confidence expressions remains underexplored. In particular, co-occurring linguistic cues, contextual variation, and subjective audience interpretation pose unique challenges. We therefore model linguistic confidence as a distribution over plausible perceived probability values that a statement is correct, capturing interpretation variability that scalar representations discard. Within this distributional framework, we introduce faithfulness as a complementary evaluation dimension and present Faithfulness Divergence (FD), an information-theoretic metric quantifying the surprise induced in audience beliefs upon truth revelation. Building on these foundations, we present Retrieval-Augmented Linguistic Calibration (RALC), a lightweight post-hoc pipeline that propagates calibrated confidence signals back into natural language via retrieval-augmented rewriting. Across three QA benchmarks and five LLM families, RALC improves in-domain faithfulness and calibration up to 66% and 58%, respectively, outperforming black-box and grey-box calibration baselines.

Problem

• linguistic calibration
• confidence expression
• audience interpretation
• contextual variation
• retrieval-augmented

Innovation

• Retrieval-Augmented Linguistic Calibration
• linguistic confidence
• faithfulness
• distributional calibration
• Faithfulness Divergence