A Task-Centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula

2026-02-10

AI Summary

This work addresses the lack of theoretical understanding regarding iterative self-improvement of large language models under limited data regimes, particularly the overlooked role of task difficulty and curriculum design. The authors model each self-improvement iteration as maximum likelihood fine-tuning on a reward-filtered data distribution and, for the first time, establish finite-sample theoretical guarantees from a task-centric perspective. Their analysis quantifies the interplay among model capacity, task difficulty, and data budget, revealing that a curriculum progressing from easy to hard tasks enables sustained performance gains and mitigates premature saturation, outperforming static task mixtures. Empirical validation through Monte Carlo simulations and graph-structured reasoning tasks confirms that this curriculum strategy significantly enhances model performance.

Technology Category

• Machine Learning: Learning Theory
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Iterative self-improvement fine-tunes an autoregressive large language model (LLM) on reward-verified outputs generated by the LLM itself. In contrast to the empirical success of self-improvement, the theoretical foundation of this generative, iterative procedure in a practical, finite-sample setting remains limited. We make progress toward this goal by modeling each round of self-improvement as maximum-likelihood fine-tuning on a reward-filtered distribution and deriving finite-sample guarantees for the expected reward. Our analysis reveals an explicit feedback loop where better models accept more data per iteration, supporting sustained self-improvement while explaining eventual saturation of such improvement. Adopting a task-centric view by considering reasoning tasks with multiple difficulty levels, we further prove quantifiable conditions on model initialization, task difficulty, and sample budget where easy-to-hard curricula provably achieve better guarantees than training on fixed mixtures of tasks. Our analyses are validated via Monte-Carlo simulations and controlled experiments on graph-based reasoning tasks.

Problem

• iterative self-improvement
• finite-sample guarantee
• easy-to-hard curriculum
• task difficulty
• large language models

Innovation

• iterative self-improvement
• easy-to-hard curriculum
• finite-sample guarantee
• task-centric theory
• reward-filtered fine-tuning