DIVE: Unlocking Self-Improvement in Frozen Language Models Through Diversity-Driven Skill Evolution

2026-08-12

AI Summary

This work addresses the challenge that frozen large language models cannot continuously learn or self-improve after deployment. To overcome this limitation, the authors propose DIVE, a novel framework that enables efficient self-improvement without parameter updates. DIVE encodes reusable reasoning skills in natural language and integrates diversity-guided skill evolution, bootstrapped experience sampling, and a teacher-free mechanism for skill execution and revision. It further introduces a multi-skill population with parallel evolution and collaborative selection strategies. The approach substantially enhances model generalization and robustness while supporting cross-model skill transfer. Evaluated on six categories of mathematical and logical reasoning tasks, DIVE achieves greater performance gains with fewer rollouts; notably, smaller models augmented via transferred skills rival or even surpass larger-scale models.

Abstract

Large language models (LLMs) cannot retain post-deployment experience without parameter updates. We introduce DIVE, a diversity-driven framework that enables frozen LLMs to improve by evolving persistent natural-language skills from task experience and verifier feedback. These skills encode reusable reasoning procedures, verification strategies, common failure modes, and output constraints and are both executed and revised by the same underlying model without access to a teacher model. Since natural-language skill evolution is a stochastic, non-convex search process, optimizing a single skill trajectory can overfit to sampled experience or converge to a suboptimal solution. DIVE mitigates this optimization variance by independently evolving multiple skill populations from bootstrapped experience, adaptively refining them through diverse transformations, and jointly selecting a complementary set of skills. Across six mathematical and logical reasoning tasks and multiple model families, DIVE consistently outperforms existing reasoning methods, prompt-optimization approaches, skill-development frameworks, and memory-based baselines. It achieves rapid self-improvement from accumulated experience, obtaining substantially larger performance gains with fewer rollouts than parameter-based methods such as SFT and GRPO, and prompt optimization with GEPA. Further, the resulting skills transfer across model scales and families, enabling smaller models such as GPT-5-nano to match or outperform larger counterparts, i.e., GPT-5, under conventional prompting. These results establish diversity-driven skill evolution as an effective, interpretable, and parameter-free approach to LLM self-improvement.

Problem

• self-improvement
• frozen language models
• skill evolution
• post-deployment learning
• parameter-free adaptation

Innovation

• Diversity-Driven Skill Evolution
• Frozen Language Models
• Natural-Language Skills
• Self-Improvement
• Parameter-Free Learning