Localizing RL-Induced Tool Use to a Single Crosscoder Feature

2026-06-24

AI Summary

This work investigates how reinforcement learning (RL) fine-tuning reshapes the internal representations of language models to enable tool use—a mechanism that remains poorly understood. The authors propose a Dedicated Feature Cross-encoder (DFC) that, for the first time, localizes RL-induced tool-use capabilities to a small set of manipulable features. Through encode-decode reconstruction and extensive validation across 48 hyperparameter configurations, they demonstrate the generality of this approach. Applied to Qwen2.5-3B, DFC improves tool-call accuracy by 31.1 ± 9.7 percentage points and enables zero-shot capability transfer: without retraining, it yields a 6.8 ± 5.0 percentage point performance gain on a frozen base model. These findings reveal that tool-use behavior is grounded in separable and transferable representational structures within the model.

Abstract

Fine-tuning through RL reshapes the internal representations of language models to enable agentic behaviors such as tool use, yet the mechanistic basis of these changes remains poorly understood. While RL substantially improves structured tool-call generation, it is unclear which features emerge, which are preserved, and whether identified features can be leveraged for retraining-free behavioral control. In this work, we show that $\textit{Dedicated Feature Crosscoders (DFC)}$ isolate a compact set of RL-specific features that mediate tool-calling capability in $\texttt{Qwen2.5-3B}$. Across a $48$-crosscoder hyperparameter sweep, encode-decode reconstruction improves the RL model's tool correctness by $+31.1 \pm {9.7}$ pp and passively transfers tool-calling ability to the frozen base model by $+6.8 \pm 5.0$ pp which we call a $\textit{capability spillover}$. Our findings show that DFC partitioning concentrates RL-introduced capability into a minimal, steerable feature set that enables runtime behavioral control of agentic LLMs.

Problem

• reinforcement learning
• tool use
• feature localization
• language models
• behavioral control

Innovation

• Dedicated Feature Crosscoders
• reinforcement learning
• tool use
• capability spillover
• behavioral control