Guided Self-Evolving LLMs with Minimal Human Supervision

2025-12-02

AI Summary

Large language models (LLMs) often suffer from concept drift, diversity collapse, and erroneous self-evolution during unsupervised or weakly supervised self-improvement, leading to performance stagnation. Method: This paper proposes R-Few—a context-anchored, guided self-play framework that couples challenger and solver agents for co-evolution, integrated with an online difficulty-curriculum-driven synthetic data generation and hybrid training paradigm. Contribution/Results: R-Few achieves stable and controllable model iteration using minimal human annotation (<5% of conventional supervised data volume). Evaluated on mathematical and general reasoning benchmarks, Qwen3-8B-Base optimized via R-Few outperforms R-Zero by +3.0 points and matches the performance of state-of-the-art models trained on 20× more human-annotated data. The framework significantly enhances both the efficiency and reliability of self-evolution in LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

AI self-evolution has long been envisioned as a path toward superintelligence, where models autonomously acquire, refine, and internalize knowledge from their own learning experiences. Yet in practice, unguided self-evolving systems often plateau quickly or even degrade as training progresses. These failures arise from issues such as concept drift, diversity collapse, and mis-evolution, as models reinforce their own biases and converge toward low-entropy behaviors. To enable models to self-evolve in a stable and controllable manner while minimizing reliance on human supervision, we introduce R-Few, a guided Self-Play Challenger-Solver framework that incorporates lightweight human oversight through in-context grounding and mixed training. At each iteration, the Challenger samples a small set of human-labeled examples to guide synthetic question generation, while the Solver jointly trains on human and synthetic examples under an online, difficulty-based curriculum. Across math and general reasoning benchmarks, R-Few achieves consistent and iterative improvements. For example, Qwen3-8B-Base improves by +3.0 points over R-Zero on math tasks and achieves performance on par with General-Reasoner, despite the latter being trained on 20 times more human data. Ablation studies confirm the complementary contributions of grounded challenger training and curriculum-based solver training, and further analysis shows that R-Few mitigates drift, yielding more stable and controllable co-evolutionary dynamics.

Problem

• Stabilizing AI self-evolution to prevent performance degradation from biases
• Reducing heavy human supervision in autonomous model learning processes
• Mitigating concept drift and diversity collapse during iterative self-training

Innovation

• Guided Self-Play Challenger-Solver framework
• Lightweight human oversight via in-context grounding
• Online difficulty-based curriculum for joint training