Reinforcement Learning without Ground-Truth Solutions can Improve LLMs

2026-06-25

AI Summary

This work addresses the limitation of traditional verifiable reward reinforcement learning, which relies on ground-truth answers and thus struggles with tasks lacking definitive solutions. The authors propose RiVER, a framework for training large language models using only scalar feedback. RiVER generates continuous supervision signals through deterministic execution and optimizes policies via group-wise relative reinforcement learning. To mitigate inconsistent scoring scales across instances and over-sampling of suboptimal solutions, the framework incorporates a calibrated reward shaping mechanism. It further integrates rank-based verifiable reinforcement learning, instance-level comparisons, top-solution emphasis, and bounded feedback retention. Experiments demonstrate that RiVER improves Qwen3-8B and GLM-Z1-9B-0414 by 8.9% and 9.4%, respectively, on ALE-Bench, and achieves average gains of 2.4% and 3.5% on exact-answer benchmarks such as LiveCodeBench and USACO.

Abstract

Reinforcement learning with verifiable rewards (RLVR) for training LLMs typically rely on ground-truth answers to assign rewards, limiting their applicability to tasks where the ground-truth solution is unknown. We introduce a \textbf{R}anking-\textbf{i}nduced \textbf{VER}ifiable framework (RiVER) that trains LLMs on score-based optimization tasks without ground-truth solutions, using deterministic execution feedback as continuous-valued supervision. When applying group-relative RL to such continuous rewards, we identify two key challenges: \emph{scale dominance}, where uncalibrated score magnitudes across test instances distort policy updates, and \emph{frequency dominance}, where repeatedly sampled suboptimal solutions can outweigh rare but stronger candidates. RiVER addresses these challenges with calibrated reward shaping that uses instance-wise comparisons and emphasizes top-ranked solvers while retaining bounded feedback for other valid solutions. We train on 12 AtCoder Heuristic Contest tasks and evaluate on Algorithm Engineering Benchmark (ALE-Bench), LiveCodeBench, and USACO. RiVER advances Qwen3-8B and GLM-Z1-9B-0414 by 8.9\% and 9.4\% in ALE rating rank. More importantly, despite training exclusively on score-based tasks without any ground-truth solutions, RiVER also improves the backbones across exact-solution benchmarks such as LiveCodeBench and USACO by an absolute average improvement of 2.4\% and 3.5\%. By contrast, baselines trained with raw execution scores improve ALE rating but fail to transfer to exact-solution benchmarks. These results suggest that score-based optimization tasks, combined with proper reward calibration, can serve as effective training environments for general coding ability without ground-truth solutions.

Problem

• Reinforcement Learning
• Ground-Truth Solutions
• Large Language Models
• Score-Based Optimization
• Reward Calibration

Innovation

• Reinforcement Learning without Ground Truth
• Reward Calibration
• Score-based Optimization
• RiVER Framework
• Execution Feedback