Likelihood-Based Reward Designs for General LLM Reasoning

2026-02-03

AI Summary

This work addresses the limitation of existing reinforcement learning approaches for fine-tuning large language models, which rely on handcrafted sparse binary rewards and struggle to generalize to long-answer reasoning tasks lacking external verifiers. The authors propose a unified training paradigm that uses the log-probability of a reference answer as a general-purpose reward signal, eliminating the need for task-specific verifiers and applicable to both verifiable and non-verifiable settings. By integrating chain-of-thought (CoT) reasoning with a likelihood-based reward function, the method achieves success rates comparable to or better than binary-reward baselines on mathematical reasoning benchmarks while significantly reducing perplexity. Moreover, in long-answer tasks without verifiers, it matches the performance of supervised fine-tuning, demonstrating superior generalization and practical utility.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Fine-tuning large language models (LLMs) on reasoning benchmarks via reinforcement learning requires a specific reward function, often binary, for each benchmark. This comes with two potential limitations: the need to design the reward, and the potentially sparse nature of binary rewards. Here, we systematically investigate rewards derived from the probability or log-probability of emitting the reference answer (or any other prompt continuation present in the data), which have the advantage of not relying on specific verifiers and being available at scale. Several recent works have advocated for the use of similar rewards (e.g., VeriFree, JEPO, RLPR, NOVER). We systematically compare variants of likelihood-based rewards with standard baselines, testing performance both on standard mathematical reasoning benchmarks, and on long-form answers where no external verifier is available. We find that using the log-probability of the reference answer as the reward for chain-of-thought (CoT) learning is the only option that performs well in all setups. This reward is also consistent with the next-token log-likelihood loss used during pretraining. In verifiable settings, log-probability rewards bring comparable or better success rates than reinforcing with standard binary rewards, and yield much better perplexity. In non-verifiable settings, they perform on par with SFT. On the other hand, methods based on probability, such as VeriFree, flatline on non-verifiable settings due to vanishing probabilities of getting the correct answer. Overall, this establishes log-probability rewards as a viable method for CoT fine-tuning, bridging the short, verifiable and long, non-verifiable answer settings.

Problem

• reward design
• reinforcement learning
• large language models
• reasoning benchmarks
• chain-of-thought

Innovation

• likelihood-based reward
• log-probability reward
• chain-of-thought reasoning
• reinforcement learning
• verifier-free