Why Summaries Turn Neutral: Policy Attribution for Sentiment Drift in Reinforcement Learning from Human Feedback

2026-08-16

AI Summary

This study addresses sentiment drift and excessive neutralization in RLHF-based summarization by proposing a strategy attribution framework that identifies the model’s low-risk preference as the underlying cause. Building on this insight, we introduce a sentiment-aware regularization technique that integrates gradient and logit decomposition to effectively mitigate drift, while also validating the cross-lingual universality of these drift patterns. Experimental results demonstrate that our approach reduces sentiment drift by 18%–22% and significantly improves sentiment fidelity without compromising summary quality. These findings establish a novel paradigm for fine-grained sentiment control during the alignment stage of large language model training.

Abstract

Reinforcement learning with human feedback (RLHF) aligns LLMs with human preferences, improving summarization fluency and safety, but causes sentiment drift: overly neutral summaries stripped of emotional nuance. We diagnose why RL acts as a sentiment neutralizer and present Policy Attribution, a framework using gradient and logit decomposition to trace drift to reward model (RM) signals and KL (Kullback-Leibler) penalty. Sentiment drift reflects a strategic bias toward "low-risk" tokens maximizing expected rewards under preference uncertainty (Stiennon et al., 2020; Gao, Schulman, and Hilton, 2023). On Reddit TL;DR and CNN/DailyMail, RLHF summaries get higher rewards but show 30-40% lower sentiment variance. Cross-lingual analysis across eight languages shows language-independent drift, with morphologically richer languages more suppressed (Krasitskii et al., 2026). We propose and validate a sentiment-aware regularization technique reducing drift by 18-22% without harming summary quality. The code and toolkit will be public.

Problem

• Sentiment Drift
• Reinforcement Learning from Human Feedback
• Summarization
• Emotional Nuance

Innovation

• Policy Attribution
• Sentiment Drift
• RLHF
• Sentiment-aware Regularization
• Gradient Decomposition