Reverse Preference Optimization for Complex Instruction Following

2025-05-28

AI Summary

Large language models (LLMs) struggle with following complex, multi-constrained instructions, and conventional preference learning suffers from coarse-grained constraint satisfaction evaluation, leading to noisy preference pairs. To address this, we propose Reverse Preference Optimization (RPO), a novel framework that dynamically constructs “perfect” positive responses satisfying all constraints via reverse constraint reasoning, enabling fine-grained alignment between model outputs and individual constraints. RPO integrates constraint-aware reverse generation, dynamic response-constraint matching, and multi-turn instruction fine-tuning within the DPO paradigm. This approach significantly widens the margin between high- and low-quality responses, reducing sampling and filtering overhead. Experiments on Sysbench and Multi-IF benchmarks show that RPO improves DPO by +4.6 and +2.5 points on Llama-3.1-8B, respectively; moreover, the Llama-3.1-70B variant surpasses GPT-4o in constrained instruction following.

Technology Category

• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Constraint Optimization
• Humans and AI: Learning Human Values and Preferences

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Instruction following (IF) is a critical capability for large language models (LLMs). However, handling complex instructions with multiple constraints remains challenging. Previous methods typically select preference pairs based on the number of constraints they satisfy, introducing noise where chosen examples may fail to follow some constraints and rejected examples may excel in certain respects over the chosen ones. To address the challenge of aligning with multiple preferences, we propose a simple yet effective method called Reverse Preference Optimization (RPO). It mitigates noise in preference pairs by dynamically reversing the constraints within the instruction to ensure the chosen response is perfect, alleviating the burden of extensive sampling and filtering to collect perfect responses. Besides, reversal also enlarges the gap between chosen and rejected responses, thereby clarifying the optimization direction and making it more robust to noise. We evaluate RPO on two multi-turn IF benchmarks, Sysbench and Multi-IF, demonstrating average improvements over the DPO baseline of 4.6 and 2.5 points (on Llama-3.1 8B), respectively. Moreover, RPO scales effectively across model sizes (8B to 70B parameters), with the 70B RPO model surpassing GPT-4o.

Problem

• Handling complex instructions with multiple constraints in LLMs
• Reducing noise in preference pairs for better alignment
• Improving robustness and performance in multi-turn instruction following

Innovation

• Reverse constraints to ensure perfect responses
• Dynamic reversal clarifies optimization direction
• Effective scaling across different model sizes