RISE-RL: Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learning

2026-08-10

AI Summary

This work addresses the challenge in open-ended reinforcement learning where large language models struggle to simultaneously satisfy multiple scoring criteria, and conventional scalar rewards obscure fine-grained feedback, thereby impeding targeted improvement of performance bottlenecks. The authors propose a criterion-aware selective exploration mechanism that identifies persistently underperforming dimensions and guides the generation of high-quality trajectories. Only samples surpassing the natural policy’s average overall score are retained, and their weakly supported behaviors are reinforced through re-evaluation with the original prompts. An auxiliary optimization objective is further introduced to dynamically modulate the guidance signal. Evaluated on both 4B and 14B models, this approach achieves state-of-the-art average scores across all benchmarks, yielding a 3.3-point average gain over standard Rubric-RL for the 14B model—with up to 6.0 points in creative writing—while enhancing output diversity and delivering significant improvements on objective tasks in domains such as medicine and science.

Abstract

Aligning Large Language Models (LLMs) for open-ended tasks is challenging because responses must satisfy multidimensional criteria without following a single correct generation trajectory. Existing rubric-based reinforcement learning (RL) methods compress fine-grained criterion-level feedback into scalar rewards, making persistent capability gaps difficult to target under limited on-policy exploration. We propose $\textbf{RISE-RL}$ (Rubric-Informed Selective Exploration), which uses repeatedly missed rubric criteria to elicit privileged trajectories that are difficult to discover through unguided exploration alone. RISE-RL retains only trajectories whose complete-rubric reward exceeds the mean reward of natural rollouts, and then re-evaluates them under the original prompt to emphasize behaviors that remain weakly supported by the natural policy. The resulting guidance signal is optimized through a separate auxiliary objective and removed once its additional benefit diminishes. Experiments with 4B and 14B models across writing, chat, health, and science show that RISE-RL achieves the highest mean score on every evaluated benchmark under guidance-free evaluation. Compared with standard Rubric-RL, it improves the average score by 1.3 points at the 4B scale and $\textbf{3.3 points at the 14B scale}$, including a $\textbf{6.0-point}$ gain on CreativeWriting-V3. It also improves creative-writing diversity and yields gains on objectively scored medical and scientific benchmarks. These results indicate that selective internalization through reward filtering and policy support shaping is effective for open-ended reinforcement learning.

Problem

• open-ended reinforcement learning
• rubric-based alignment
• large language models
• multidimensional criteria
• capability gaps

Innovation

• Selective Exploration
• Rubric-Based Reinforcement Learning
• Privileged Trajectories
• Reward Filtering
• Policy Support Shaping