IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards

2025-08-06

AI Summary

To address the inefficiency and reward hacking issues—such as exploiting validation shortcuts and deviating from users’ true intent—in Reinforcement Learning from Verification Rewards (RLVR) for enhancing LLM instruction-following capability, this paper proposes IntentGuard. Our method comprises three core components: (1) a collaborative adversarial data flywheel that automatically generates progressively challenging instructions; (2) an IntentCheck module that explicitly models and constrains alignment with user intent; and (3) diagnostic “trip-wire” instructions to detect and suppress reward-hacking behaviors in real time. IntentGuard integrates hybrid verifiable rewards, adversarial instruction construction, and intent-aware verification. Evaluated on Qwen2.5-32B-Instruct, it achieves 87.43% accuracy on IFEval—surpassing GPT-4o—and yields significant gains on FollowBench, while substantially reducing reward-hacking incidence.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) improves instruction following capabilities of large language models (LLMs), but suffers from training inefficiency due to inadequate difficulty assessment. Moreover, RLVR is prone to over-optimization, where LLMs exploit verification shortcuts without aligning to the actual intent of user instructions. We introduce Instruction Following Decorator (IFDecorator}, a framework that wraps RLVR training into a robust and sample-efficient pipeline. It consists of three components: (1) a cooperative-adversarial data flywheel that co-evolves instructions and hybrid verifications, generating progressively more challenging instruction-verification pairs; (2) IntentCheck, a bypass module enforcing intent alignment; and (3) trip wires, a diagnostic mechanism that detects reward hacking via trap instructions, which trigger and capture shortcut exploitation behaviors. Our Qwen2.5-32B-Instruct-IFDecorator achieves 87.43% accuracy on IFEval, outperforming larger proprietary models such as GPT-4o. Additionally, we demonstrate substantial improvements on FollowBench while preserving general capabilities. Our trip wires show significant reductions in reward hacking rates. We will release models, code, and data for future research.

Problem

• Improving training efficiency in RLVR for LLMs
• Preventing over-optimization and intent misalignment in RLVR
• Detecting and reducing reward hacking in instruction following

Innovation

• Cooperative-adversarial data flywheel for evolving challenges
• IntentCheck module ensures instruction intent alignment
• Trip wires detect and reduce reward hacking