Recovering Diversity Without Losing Alignment: A DPO Recipe for Post-Trained LLMs

2026-05-28

AI Summary

This work addresses the significant decline in output diversity observed in large language models after instruction tuning. To mitigate this issue, the authors propose REDIPO, a method that samples responses from both the base and instruction-tuned models, rewrites them, and filters for safe and compliant candidates to construct offline preference pairs that explicitly encourage marginal diversity. These pairs are then integrated into Direct Preference Optimization (DPO) to recover diversity without compromising alignment performance. Evaluated on Qwen3-4B, OLMo-3-7B, and LLaMA-3.1-8B, REDIPO improves distinct_k scores on NoveltyBench by 134%, 33%, and 44%, respectively, while preserving competitive performance on MTBench, IFEval, and Arena-Hard, and simultaneously reducing attack success rates on HarmBench.

Abstract

Many open-ended instructions have multiple valid answers that users can benefit from seeing, but post-training often narrows an LLM's output space toward a small set of canonical responses. We introduce REDIPO, an offline DPO data-construction pipeline for recovering distinct valid answer modes while preserving the alignment benefits of the instruct model. For each prompt, REDIPO samples responses from both base and instruct models, rewrites base-model responses with the instruct model, filters candidates for safety and instruction-following quality, and builds preference pairs that favor marginally diverse responses among candidates with similar instruction-following reward. Across Qwen3-4B, OLMo-3-7B, and LLaMA-3.1-8B, REDIPO improves NoveltyBench distinct_k by 134%, 33%, and 44% relative to the instruct checkpoints, while DivPO changes diversity by 0%, -6%, and -4% on the same models. These gains largely maintain MTBench, IFEval, and Arena-Hard performance, and reduce direct-category HarmBench attack success rate. Ablations show that marginal-diversity pair selection and base-response rewriting drive the diversity gains, while filtering and quality-bounded pairing help maintain alignment. Overall, our results show that diverse valid answers from base-model generations can be reintroduced through carefully constructed preference data while retaining the alignment benefits of post-training. We release our code and data at https://github.com/vsamuel2003/RiDiPO.

Problem

• diversity
• alignment
• large language models
• instruction tuning
• response generation

Innovation

• DPO
• response diversity
• preference optimization
• instruction alignment
• base-model rewriting