Geometry-Calibrated Conformal Abstention for Language Models

2026-04-30

AI Summary

This work addresses the tendency of language models to produce plausible yet incorrect responses when lacking relevant knowledge. To mitigate this issue, the authors propose a Conformal Abstention framework that extends conformal prediction to open-ended generation settings. By analyzing the geometric structure of internal model representations, the method calibrates prediction confidence to more accurately detect states of model ignorance and determine when to abstain from answering. This approach enables controllable modulation of response behavior, significantly improving performance on selective question-answering tasks. Empirical results demonstrate a conditional accuracy of 75%, effectively balancing answer coverage and correctness.

Abstract

When language models lack relevant knowledge for a given query, they frequently generate plausible responses that can be hallucinations, rather than admitting being agnostic about the answer. Retraining models to reward admitting ignorance can lead to overly conservative behaviors and poor generalization due to scarce evaluation benchmarks. We propose a post hoc framework, Conformal Abstention (CA), adapted from conformal prediction (CP) to determine whether to abstain from answering a query. CA provides finite-sample guarantees on both the probability of participation (i.e., not abstaining) and the probability that the generated response is correct. Importantly, the abstention decision relies on prediction confidence rather than the non-conformity scores used in CP, which are intractable for open-ended generation. To better align prediction confidence with the model's ignorance, we introduce a calibration strategy using representation geometry within the model to measure knowledge involvement in shaping the response. Experiments demonstrate that we improve selective answering significantly with 75 percent conditional correctness.

Problem

• language models
• hallucination
• abstention
• conformal prediction
• knowledge calibration

Innovation

• Conformal Abstention
• prediction confidence
• representation geometry
• selective answering
• hallucination mitigation