Maximizing Confidence Alone Improves Reasoning

2025-05-28

AI Summary

Large language models (LLMs) struggle to improve complex reasoning capabilities without external reward signals or ground-truth supervision. Method: We propose RENT, a reinforcement learning framework that performs end-to-end entropy minimization using only the entropy of the model’s own output distribution as an intrinsic reward signal. RENT fully replaces handcrafted reward functions or human-annotated answers with self-assessed confidence (i.e., output entropy), and integrates chain-of-thought distillation with unsupervised policy updates to achieve purely self-supervised reasoning enhancement. Contribution/Results: Evaluated across diverse benchmarks—including GSM8K, MATH500, AMC, AIME, and GPQA—RENT consistently improves mathematical and scientific reasoning performance for Qwen and Mistral series models. It demonstrates, for the first time, that complex reasoning can be effectively enhanced without any external supervision, establishing both feasibility and cross-domain generalizability of pure self-supervision in reasoning optimization.

Technology Category

• Machine Learning: Reinforcement Learning
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Reinforcement learning (RL) has enabled machine learning models to achieve significant advances in many fields. Most recently, RL has empowered frontier language models to solve challenging math, science, and coding problems. However, central to any RL algorithm is the reward function, and reward engineering is a notoriously difficult problem in any domain. In this paper, we propose RENT: Reinforcement Learning via Entropy Minimization -- a fully unsupervised RL method that requires no external reward or ground-truth answers, and instead uses the model's entropy of its underlying distribution as an intrinsic reward. We find that by reinforcing the chains of thought that yield high model confidence on its generated answers, the model improves its reasoning ability. In our experiments, we showcase these improvements on an extensive suite of commonly-used reasoning benchmarks, including GSM8K, MATH500, AMC, AIME, and GPQA, and models of varying sizes from the Qwen and Mistral families. The generality of our unsupervised learning method lends itself to applicability in a wide range of domains where external supervision is unavailable.

Problem

• Unsupervised RL method for reasoning without external rewards
• Improves reasoning by maximizing model confidence via entropy minimization
• Applicable in domains with limited or no external supervision

Innovation

• Unsupervised RL via entropy minimization
• Intrinsic reward from model confidence
• Improves reasoning without external rewards