DIVE: Diversified Iterative Self-Improvement

2025-01-01

AI Summary

Large language models (LLMs) undergoing iterative self-improvement (ISI) suffer from deteriorating output diversity due to repeated learning from their own generations—particularly detrimental in multi-step mathematical reasoning tasks requiring diverse solution paths. To address this, we propose a diversity-quality co-optimized sample pool expansion and filtering framework: it dynamically expands the sampling space, constructs preference pairs, and incorporates a diversity-aware selection strategy to overcome the homogenization bottleneck inherent in conventional ISI methods. Evaluated on MATH and GSM8K, our approach achieves 10–45% relative improvement in output diversity while strictly preserving reasoning accuracy. The core contribution lies in the first explicit integration of diversity modeling across the entire ISI data construction and filtering pipeline—establishing a novel paradigm for controllable, robust model self-evolution.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advances in large language models (LLMs) have demonstrated the effectiveness of Iterative Self-Improvement (ISI) techniques. However, continuous training on self-generated data leads to reduced output diversity, a limitation particularly critical in reasoning tasks where diverse solution paths are essential. We present DIVE (Diversified Iterative Self-Improvement), a novel framework that addresses this challenge through two key components: Sample Pool Expansion for broader solution exploration, and Data Selection for balancing diversity and quality in preference pairs. Experiments on MATH and GSM8k datasets show that DIVE achieves a 10% to 45% relative increase in output diversity metrics while maintaining performance quality compared to vanilla ISI. Our ablation studies confirm both components' significance in achieving these improvements. Code is available at https://github.com/qinyiwei/DIVE.

Problem

• Large Language Models
• Self-Learning
• Diversity Reduction

Innovation

• DIVE
• Iterative Self-Improvement
• Diversity Enhancement