In LLM Reasoning, there is Irrationality on top of Value Misalignment

2026-05-26

arXiv.org

AI Summary

论文探讨了即使对齐目标价值函数，大语言模型在推理中仍可能无法最大化该价值的问题，并通过数学形式化分析和实验验证了解决方法的有效性。

Abstract

Significant progress has been made in aligning LLMs with target value functions. We argue that, even when an LLM has been well aligned in (post-)training, it may still fail to maximise the aligned value in reasoning. We mathematically formalise this gap as rational value risk: the utility discrepancy between a model's deployed reasoning strategy and its rational counterpart, which is defined to be the responses that maximise expected utility in the steepest direction. The estimation error of rational value risk is further decomposed into three components from finite candidates, finite prompts, and imperfect verifiers. Extensive experiments are conducted, covering models Llama-3.1, Qwen-2.5, T{\"}ulu-3 families (7B-72B), GPT-5.2, GPT-5.5, and DeepSeek-V4, and benchmarks UltraFeedback, AlpacaEval, GSM8K, MATH, HumanEval, and MathArena. The results validate that (1) rational value risk is widespread; (2) value alignment can reduce, but cannot eliminate, it; (3) the risk is highly sensitive to inference-time reasoning strategy; and (4) longer reasoning improves rationality with diminishing returns. The code is at https://github.com/EVIEHub/LLM-Rationality.

Problem

• LLM Reasoning
• Value Misalignment
• Rational Value Risk

Innovation

• rational value risk
• value alignment
• self-consistency
• chain of thought