Learning to Self-Evolve

2026-03-19

AI Summary

Current large language models lack explicit training during inference to autonomously refine their contextual representations, limiting their ability to leverage feedback for improved performance on novel tasks. This work proposes a reinforcement learning framework that, for the first time, formulates self-evolution as a learnable skill. By introducing a tree-guided context editing mechanism, the approach enables multi-step self-evolution driven by single-step rewards at test time. This paradigm shifts beyond reliance on a model’s intrinsic reasoning capabilities and demonstrates transferable guidance across different architectures. Empirical results show that a 4B-parameter model equipped with this method surpasses advanced self-evolution strategies—including those of GPT-5 and Claude Sonnet 4.5—as well as prompt optimization techniques such as GEPA and TextGrad on challenging benchmarks like Text-to-SQL (BIRD) and question answering (MMLU-Redux).

Technology Category

• Machine Learning: Evolutionary Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We introduce Learning to Self-Evolve (LSE), a reinforcement learning framework that trains large language models (LLMs) to improve their own contexts at test time. We situate LSE in the setting of test-time self-evolution, where a model iteratively refines its context from feedback on seen problems to perform better on new ones. Existing approaches rely entirely on the inherent reasoning ability of the model and never explicitly train it for this task. LSE reduces the multi-step evolution problem to a single-step RL objective, where each context edit is rewarded by the improvement in downstream performance. We pair this objective with a tree-guided evolution loop. On Text-to-SQL generation (BIRD) and general question answering (MMLU-Redux), a 4B-parameter model trained with LSE outperforms self-evolving policies powered by GPT-5 and Claude Sonnet 4.5, as well as prompt optimization methods including GEPA and TextGrad, and transfers to guide other models without additional training. Our results highlight the effectiveness of treating self-evolution as a learnable skill.

Problem

• self-evolution
• test-time adaptation
• context refinement
• reinforcement learning
• large language models

Innovation

• Learning to Self-Evolve
• test-time self-evolution
• reinforcement learning
• context editing
• prompt optimization