CoRA: Confidence-Rationale Alignment for Reliable Chain-of-Thought Reasoning

2026-06-12

AI Summary

Large language models often generate highly confident yet incorrect answers in chain-of-thought reasoning due to incomplete or flawed reasoning rationales. This work proposes a GRPO-based reinforcement learning framework that, for the first time, systematically defines and jointly optimizes answer correctness, confidence calibration, and rationale quality. The approach introduces reference-free, multi-dimensional scoring rules to align model confidence with the fidelity of the underlying reasoning process. Experimental results on benchmark datasets—including MedQA, MathQA, and OpenBookQA—demonstrate that the proposed method reduces confidence–rationale misalignment error by up to 26.51% compared to supervised fine-tuning and GRPO variants that optimize correctness alone, while simultaneously maintaining strong accuracy and calibration performance.

Abstract

Chain-of-thought (CoT) reasoning can improve LLM performance, but high answer confidence may be misleading when the accompanying CoT rationale is plausible yet incomplete or poorly supported. We study confidence--rationale alignment: whether a model's confidence in its committed answer is justified by its generated rationale. We introduce a GRPO-based reinforcement learning framework that jointly rewards answer correctness, committed-answer probability, and rubric-based rationale support, where the rubric assesses grounding, coherence, task match, and connection to the selected answer without revealing the gold answer to the judge. Across MedQA, MathQA, and OpenBookQA using three open-weight LLMs, our method reduces the confidence--rationale alignment error by up to 26.51% compared with untuned checkpoints, SFT, and correctness-only GRPO, while maintaining competitive accuracy and often improving calibration. These results show that reliable CoT reasoning requires not only confident answers, but rationales that substantively support them.

Problem

• Chain-of-Thought
• confidence alignment
• rationale
• reliability
• large language models

Innovation

• Confidence-Rationale Alignment
• Chain-of-Thought Reasoning
• GRPO
• Rationale Evaluation Rubric
• LLM Calibration