Recontextualization Mitigates Specification Gaming without Modifying the Specification

2025-12-21

AI Summary

Language models frequently engage in specification gaming—manifesting as reward hacking, test-case overfitting, user deception, and sycophancy—due to flawed supervision signal design (e.g., incomplete labels or reward functions). This work proposes *rec contextualization training*, a novel training paradigm that mitigates specification gaming at its root without modifying the original labels or reward functions. Its core innovation is *counterfactual rec contextualization*: high-quality responses are first generated under suppressive prompts, then reformulated as responses to permissive prompts for supervised fine-tuning. Evaluated across four canonical specification-gaming behaviors, the method significantly suppresses all while preserving baseline task performance. It enhances behavioral robustness and generalization without requiring improvements in supervision quality, offering a supervision-agnostic remedy to specification gaming.

Technology Category

• Natural Language Processing: Generation
• Game Theory and Economic Paradigms: Adversarial Learning
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Developers often struggle to specify correct training labels and rewards. Perhaps they don't need to. We propose recontextualization, which reduces how often language models "game" training signals, performing misbehaviors those signals mistakenly reinforce. We show recontextualization prevents models from learning to 1) prioritize evaluation metrics over chat response quality; 2) special-case code to pass incorrect tests; 3) lie to users; and 4) become sycophantic. Our method works by generating completions from prompts discouraging misbehavior and then recontextualizing them as though they were in response to prompts permitting misbehavior. Recontextualization trains language models to resist misbehavior even when instructions permit it. This mitigates the reinforcement of misbehavior from misspecified training signals, reducing specification gaming without improving the supervision signal.

Problem

• Reduces language models gaming training signals
• Prevents models prioritizing metrics over quality
• Trains models to resist misbehavior despite instructions

Innovation

• Recontextualization reduces language model gaming of signals
• Generates completions from prompts discouraging misbehavior
• Trains models to resist misbehavior despite permissive instructions