Online Reasoning Calibration: Test-Time Training Enables Generalizable Conformal LLM Reasoning

2026-04-01

AI Summary

Large language models often suffer from poor calibration during inference, leading to high computational overhead and limited generalization. This work proposes ORCA, an online inference calibration framework that uniquely integrates conformal prediction with test-time training. ORCA employs meta-learning to dynamically update a calibration module for each input and leverages self-consistency labels to enable efficient confidence estimation. The method provides theoretical risk control while substantially improving computational efficiency: under a risk tolerance δ=0.1, it reduces computational cost by 47.5% on in-domain tasks for Qwen2.5-32B; on the zero-shot out-of-domain MATH-500 benchmark, the savings increase from 24.8% to 67.0%, all while maintaining low error rates.

Abstract

While test-time scaling has enabled large language models to solve highly difficult tasks, state-of-the-art results come at exorbitant compute costs. These inefficiencies can be attributed to the miscalibration of post-trained language models, and the lack of calibration in popular sampling techniques. Here, we present Online Reasoning Calibration (ORCA), a framework for calibrating the sampling process that draws upon conformal prediction and test-time training. Specifically, we introduce a meta-learning procedure that updates the calibration module for each input. This allows us to provide valid confidence estimates under distributional shift, e.g. in thought patterns that occur across different stages of reasoning, or in prompt distributions between model development and deployment. ORCA not only provides theoretical guarantees on conformal risks, but also empirically shows higher efficiency and generalization across different reasoning tasks. At risk level $δ=0.1$, ORCA improves Qwen2.5-32B efficiency on in-distribution tasks with savings up to 47.5% with supervised labels and 40.7% with self-consistency labels. Under zero-shot out-of-domain settings, it improves MATH-500 savings from 24.8% of the static calibration baseline to 67.0% while maintaining a low empirical error rate, and the same trend holds across model families and downstream benchmarks. Our code is publicly available at https://github.com/wzekai99/ORCA.

Problem

• calibration
• large language models
• test-time training
• distributional shift
• conformal prediction

Innovation

• Online Reasoning Calibration
• Conformal Prediction
• Test-Time Training
• Distributional Shift
• Efficiency-Accuracy Tradeoff