When Can LLMs Learn to Reason with Weak Supervision?

2026-04-20

AI Summary

This work investigates whether large language models can achieve effective reasoning generalization under weak supervision—specifically in settings characterized by data scarcity, noisy rewards, or self-supervised proxy rewards—through reinforcement learning. Through a systematic analysis of training dynamics across multiple models and tasks, we identify reward saturation as a critical factor governing generalization performance and introduce "reasoning faithfulness" as a predictive metric for generalization. By integrating reinforcement learning with verifiable rewards (RLVR), supervised fine-tuning (SFT), and continued pretraining, we demonstrate successful reasoning generalization in all three weakly supervised scenarios on Llama3.2-3B-Base, substantially outperforming baseline approaches. Our findings also disentangle the distinct roles of continued pretraining and SFT in facilitating this generalization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Reasoning with Beliefs

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models have achieved significant reasoning improvements through reinforcement learning with verifiable rewards (RLVR). Yet as model capabilities grow, constructing high-quality reward signals becomes increasingly difficult, making it essential to understand when RLVR can succeed under weaker forms of supervision. We conduct a systematic empirical study across diverse model families and reasoning domains under three weak supervision settings: scarce data, noisy rewards, and self-supervised proxy rewards. We find that generalization is governed by training reward saturation dynamics: models that generalize exhibit a prolonged pre-saturation phase during which training reward and downstream performance climb together, while models that saturate rapidly memorize rather than learn. We identify reasoning faithfulness, defined as the extent to which intermediate steps logically support the final answer, as the pre-RL property that predicts which regime a model falls into, while output diversity alone is uninformative. Motivated by these findings, we disentangle the contributions of continual pre-training and supervised fine-tuning, finding that SFT on explicit reasoning traces is necessary for generalization under weak supervision, while continual pre-training on domain data amplifies the effect. Applied together to Llama3.2-3B-Base, these interventions enable generalization across all three settings where the base model previously failed.

Problem

• weak supervision
• reasoning
• reinforcement learning
• generalization
• reward signals

Innovation

• weak supervision
• reasoning faithfulness
• reward saturation
• supervised fine-tuning
• reinforcement learning with verifiable rewards