OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning

2026-06-24

AI Summary

This work addresses the instability in reinforcement learning training for open-ended tasks—such as creative writing—caused by style bias and positional inconsistency in existing large language model–based reward mechanisms. The authors propose OPERA, a novel approach that discards unreliable external critics and instead introduces an intrinsic reward mechanism grounded in perplexity dynamics, guiding alignment through reduced uncertainty during reflective states. OPERA integrates prompt-guided synthesis of diverse reasoning trajectories, rollout prioritization based on internal log-probabilities, and a cold-start data filtering pipeline. Evaluated on a newly curated dataset of 20,000 high-quality reasoning trajectories, OPERA achieves a new state-of-the-art among open-source models on Qwen3-8B, matching or even surpassing the performance of proprietary systems such as Gemini2.5 and MiniMax-M2.5 on several open-ended tasks.

Abstract

Reinforcement Learning (RL) has enabled LLMs to excel in objective reasoning tasks such as mathematics and code generation. However, applying RL to open-ended tasks, such as creative writing, remains challenging because LLM-as-a-judge reward models often exhibit stylistic biases and positional inconsistencies, leading to unstable supervision. To address this, we propose OPERA (Objective Perplexity-based Reflective Alignment), which replaces unreliable external judges with intrinsic rewards derived from perplexity dynamics. Specifically, we derive an intrinsic reward signal from perplexity dynamics, quantifying uncertainty reduction at critical reflective states. During the cold-start phase, we introduce a data synthesis method that leverages carefully designed guiding words to generate diverse reasoning traces, along with perplexity-prioritized rollouts that utilize internal log-probabilities to identify logically consistent reasoning branches. This pipeline yields a large-scale dataset comprising 20,000 high-quality reasoning trajectories. Empirical evaluations consistently demonstrate the scalability and efficacy of our approach in alignment for open-ended tasks. Implementing OPERA on Qwen3-8B establishes a new state-of-the-art among open-source models, achieving parity with or surpassing proprietary models like Gemini2.5 and MiniMax-M2.5 in some open-ended tasks. The code is available at https://github.com/pangpang-xuan/OPERA.

Problem

• open-ended reasoning
• reinforcement learning
• reward model bias
• alignment
• perplexity

Innovation

• perplexity-based reward
• open-ended reasoning
• intrinsic reward
• reflective alignment
• reinforcement learning