In-Context Reinforcement Learning for Tool Use in Large Language Models

2026-03-09

AI Summary

This work proposes a purely reinforcement learning (RL)–driven training framework for large language models to overcome their limitations in complex tasks stemming from reliance on internal knowledge and ineffective external tool invocation. Departing from conventional supervised fine-tuning (SFT), the approach introduces a small number of in-context examples during rollouts to guide tool usage and employs a progressive example annealing strategy, ultimately enabling zero-shot tool utilization. By integrating contextual reinforcement learning with standard RL algorithms, the method achieves state-of-the-art performance across multiple reasoning and tool-use benchmarks. Notably, it demonstrates for the first time that highly effective and scalable tool-calling capabilities can be trained without any supervised fine-tuning.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Reinforcement Learning
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

While large language models (LLMs) exhibit strong reasoning abilities, their performance on complex tasks is often constrained by the limitations of their internal knowledge. A compelling approach to overcome this challenge is to augment these models with external tools -- such as Python interpreters for mathematical computations or search engines for retrieving factual information. However, enabling models to use these tools effectively remains a significant challenge. Existing methods typically rely on cold-start pipelines that begin with supervised fine-tuning (SFT), followed by reinforcement learning (RL). These approaches often require substantial amounts of labeled data for SFT, which is expensive to annotate or synthesize. In this work, we propose In-Context Reinforcement Learning (ICRL), an RL-only framework that eliminates the need for SFT by leveraging few-shot prompting during the rollout stage of RL. Specifically, ICRL introduces in-context examples within the rollout prompts to teach the model how to invoke external tools. Furthermore, as training progresses, the number of in-context examples is gradually reduced, eventually reaching a zero-shot setting where the model learns to call tools independently. We conduct extensive experiments across a range of reasoning and tool-use benchmarks. Results show that ICRL achieves state-of-the-art performance, demonstrating its effectiveness as a scalable, data-efficient alternative to traditional SFT-based pipelines.

Problem

• tool use
• large language models
• reinforcement learning
• supervised fine-tuning
• in-context learning

Innovation

• In-Context Learning
• Reinforcement Learning
• Tool Use
• Large Language Models
• Few-Shot Prompting