Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models

2026-04-27

AI Summary

This study addresses why reinforcement learning (RL) fine-tuning enhances the cross-domain reasoning capabilities of large language models, whereas supervised fine-tuning (SFT) often leads to catastrophic forgetting of general abilities. Through controlled experiments, the authors align and analyze the internal activations induced by RL and SFT within a shared feature space. They reveal, for the first time at the representational level, that RL promotes generalization by preserving foundational representations and continuously refining a compact set of task-agnostic features that causally drive broad applicability. In contrast, SFT rapidly overwrites these with specialized, narrow features. Using interpretability techniques—including feature alignment, activation analysis, and targeted interventions such as feature ablation or amplification—the authors empirically demonstrate the causal role of this RL-evolved feature set in enabling robust generalization.

Abstract

Reinforcement learning (RL)-based post-training often improves the reasoning performance of large language models (LLMs) beyond the training domain, while supervised fine-tuning (SFT) frequently leads to general capabilities forgetting. However, the mechanisms underlying this contrast remain unclear. To bridge this gap, we present a feature-level mechanistic analysis methodology to probe RL generalization using a controlled experimental setup, where RL- and SFT-tuned models are trained from the same base model on identical data. Leveraging our interpretability framework, we align internal activations across models within a shared feature space and analyze how features evolve during post-training. We find that SFT rapidly introduces many highly specialized features that stabilize early in training, whereas RL induces more restrained and continually evolving feature changes that largely preserve base models' representations. Focusing on samples where RL succeeds but the base model fails, we identify a compact, task-agnostic set of features that directly mediate generalization across diverse tasks. Feature-level interventions confirm their causal role: disabling these features significantly degrades RL models' generalization performance, while amplifying them improves base models' performance. The code is available at https://github.com/danshi777/RL-generalization.

Problem

• reinforcement learning
• generalization
• large language models
• supervised fine-tuning
• feature-level analysis

Innovation

• reinforcement learning
• feature-level analysis
• model generalization
• post-training
• interpretability