Triviality Corrected Endogenous Reward

2026-04-13

AI Summary

Open-ended text generation lacks verifiable reward signals, and existing intrinsic reward methods often suffer from a "triviality bias," favoring high-probability yet unremarkable outputs. This work proposes TCER, the first approach to explicitly identify and mitigate this issue by leveraging the relative information gain between a general reference policy and a task-specific policy, augmented with a probability-dependent correction mechanism to construct an unsupervised intrinsic reward. Built upon confidence modulation, reference-policy contrastive learning, and an unsupervised reinforcement learning framework, TCER consistently improves performance across multiple writing benchmarks and diverse model architectures. Furthermore, it successfully transfers to mathematical reasoning tasks, demonstrating its generality and effectiveness without requiring external supervision.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Generation
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Reinforcement learning for open-ended text generation is constrained by the lack of verifiable rewards, necessitating reliance on judge models that require either annotated data or powerful closed-source models. Inspired by recent work on unsupervised reinforcement learning for mathematical reasoning using confidence-based endogenous rewards, we investigate whether this principle can be adapted to open-ended writing tasks. We find that directly applying confidence rewards leads to Triviality Bias: the policy collapses toward high-probability outputs, reducing diversity and meaningful content. We propose TCER (Triviality Corrected Endogenous Reward), which addresses this bias by rewarding the relative information gain between a specialist policy and a generalist reference policy, modulated by a probability-dependent correction mechanism. Across multiple writing benchmarks and model architectures, TCER achieves consistent improvements without external supervision. Furthermore, TCER also transfers effectively to mathematical reasoning, validating the generality of our approach across different generation tasks.

Problem

• open-ended text generation
• reinforcement learning
• endogenous reward
• triviality bias
• reward sparsity

Innovation

• Endogenous Reward
• Triviality Bias
• Reinforcement Learning
• Open-ended Text Generation
• Information Gain