Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability

2026-01-26

AI Summary

This work addresses the challenge that large language models struggle to improve on reasoning tasks with extremely low initial success rates—such as difficult mathematical problems, where baseline performance can be as low as 0/128—due to the absence of meaningful learning signals. To overcome this, the authors propose the SOAR framework, which introduces the first two-level meta-reinforcement learning approach grounded in genuine learning progress. In SOAR, a teacher model autonomously generates synthetic problems to form a curriculum, using the student model’s actual improvement on a challenging subset as the reward signal. Notably, the teacher does not require prior problem-solving capability, and the framework prioritizes the structural quality of generated problems over answer correctness. This design effectively mitigates the instability and diversity collapse commonly observed in traditional self-play methods, substantially enhancing the model’s reasoning capabilities.

Technology Category

• Search and Optimization: Metareasoning and Metaheuristics
• Machine Learning: Reinforcement Learning
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Can a model learn to escape its own learning plateau? Reinforcement learning methods for finetuning large reasoning models stall on datasets with low initial success rates, and thus little training signal. We investigate a fundamental question: Can a pretrained LLM leverage latent knowledge to generate an automated curriculum for problems it cannot solve? To explore this, we design SOAR: A self-improvement framework designed to surface these pedagogical signals through meta-RL. A teacher copy of the model proposes synthetic problems for a student copy, and is rewarded with its improvement on a small subset of hard problems. Critically, SOAR grounds the curriculum in measured student progress rather than intrinsic proxy rewards. Our study on the hardest subsets of mathematical benchmarks (0/128 success) reveals three core findings. First, we show that it is possible to realize bi-level meta-RL that unlocks learning under sparse, binary rewards by sharpening a latent capacity of pretrained models to generate useful stepping stones. Second, grounded rewards outperform intrinsic reward schemes used in prior LLM self-play, reliably avoiding the instability and diversity collapse modes they typically exhibit. Third, analyzing the generated questions reveals that structural quality and well-posedness are more critical for learning progress than solution correctness. Our results suggest that the ability to generate useful stepping stones does not require the preexisting ability to actually solve the hard problems, paving a principled path to escape reasoning plateaus without additional curated data.

Problem

• reasoning plateau
• sparse rewards
• automated curriculum
• self-improvement
• learnability

Innovation

• self-improvement
• meta-reinforcement learning
• automated curriculum generation
• reasoning plateau
• large language models