Beyond Solvability: Task Learnability as a Static Prior for LLM RL Post-Training

2026-08-10

AI Summary

This work addresses a key limitation in current reinforcement learning–based post-training of large language models: existing task sampling strategies overlook the varying degrees to which tasks respond positively to optimization, relying solely on immediate solvability as a proxy for long-term utility. To remedy this, the authors introduce the concept of “task learnability,” defined through the expected reward trajectory of a task under a fixed RL mechanism. They propose TrajVal, a lightweight probing method that statically estimates this property prior to training to inform task sampling. As the first approach to incorporate task learnability as a static prior in LLM post-training, TrajVal demonstrates strong reproducibility and predictability of downstream performance, and can be deployed either independently or in conjunction with online scheduling. Experiments show that TrajVal significantly improves data efficiency across mathematical and logical reasoning tasks on models of varying scales and yields complementary gains when combined with existing methods.

Abstract

Reinforcement learning (RL) has become a central post-training paradigm for eliciting reasoning capabilities in large language models, yet uniform task sampling allocates compute without regard to differences in how tasks respond to optimization. Existing task-valuation methods mostly rely on snapshot-based signals such as current pass rate or reward, which estimate how solvable a task is under the current policy. However, tasks with similar current solvability can still differ substantially in how positively they respond to further training. We study this residual axis as task learnability: a regime-conditional measure of expected positive response to continued training under a fixed RL post-training regime. By analyzing per-task reward trajectories, we find that learnability is reproducible across independently sampled training contexts and predictive of downstream utility. To make this signal practical before training begins, we propose TrajVal, a lightweight probe-based estimator that approximates per-task learnability from a short probe run and two endpoint evaluations. TrajVal can be used either as a standalone static prior for task sampling or as a multiplicative prior for existing online schedulers. Experiments on mathematical and logical reasoning benchmarks across multiple model scales show that TrajVal improves data efficiency over uniform sampling and provides complementary gains when combined with online scheduling methods.

Problem

• task learnability
• reinforcement learning
• LLM post-training
• task sampling
• data efficiency

Innovation

• task learnability
• reinforcement learning post-training
• TrajVal
• static prior
• data efficiency