Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes

2026-04-20

AI Summary

This work investigates how to efficiently fine-tune small language models under constraints of limited labeled data and computational resources by leveraging reinforcement learning with verifiable rewards (RLVR). To enable precise control over data scale, diversity, and complexity, the authors construct three procedurally generated datasets encompassing numerical counting, graph reasoning, and spatial reasoning tasks. Experimental results demonstrate that, in low-data regimes, training on a mixture of tasks with varying complexity yields up to a fivefold improvement in sample efficiency compared to training solely on simple tasks, while also significantly enhancing the model’s ability to generalize from simpler to more complex tasks. This study establishes a controllable and reproducible paradigm for efficient model fine-tuning in resource-constrained settings.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Fine-tuning Large Language Models (LLMs) typically relies on large quantities of high-quality annotated data, or questions with well-defined ground truth answers in the case of Reinforcement Learning with Verifiable Rewards (RLVR). While previous work has explored the benefits to model reasoning capabilities by scaling both data and compute used for RLVR, these results lack applicability in many real-world settings where annotated data and accessible compute may be scarce. In this work, we present a comprehensive empirical study of open-source Small Language Model (SLM) performance after RLVR in low data regimes. Across three novel datasets covering number counting problems, graph reasoning, and spatial reasoning, we characterize how model performance scales with dataset size, diversity, and complexity. We demonstrate that (1) procedural datasets allow for fine-grained evaluation and training dataset development with controllable properties (size, diversity, and complexity), (2) under RLVR, models trained on lower complexity tasks can generalize to higher complexity tasks, and (3) training on mixed complexity datasets is associated with the greatest benefits in low data regimes, providing up to 5x sample efficiency versus training on easy tasks. These findings inspire future work on the development of data scaling laws for RLVR and the use of procedural data generators to further understand effective data development for efficient LLM fine-tuning.

Problem

• RLVR
• low data regimes
• Small Language Models
• data efficiency
• fine-tuning

Innovation

• RLVR
• procedural datasets
• sample efficiency
• low-data regime
• Small Language Models