E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoning

2026-04-10

AI Summary

This work addresses key challenges in tool-integrated reasoning with large language models, including inefficient exploration, high data costs, and performance plateaus. The authors propose E3-TIR, a method that dynamically integrates three types of experience—expert prefixes, expert-guided trajectories, and autonomous exploration—during early training to establish a diverse exploration mechanism centered around expert anchors. This approach effectively mitigates distributional shift and optimization conflicts. By combining branching exploration with hybrid policy optimization, E3-TIR achieves a 6% improvement in tool-use task performance while using less than 10% of synthetic data compared to baselines. Furthermore, it yields a 1.46× higher return on investment (ROI), substantially enhancing both training efficiency and downstream reasoning capabilities.

Technology Category

• Search and Optimization: Metareasoning and Metaheuristics
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

While Large Language Models (LLMs) have demonstrated significant potential in Tool-Integrated Reasoning (TIR), existing training paradigms face significant limitations: Zero-RL suffers from inefficient exploration and mode degradation due to a lack of prior guidance, while SFT-then-RL is limited by high data costs and capability plateaus caused by low-entropy collapse. To address these challenges, we propose E3-TIR (Enhanced Experience Exploitation), a warm-up paradigm for the early stages of agent training. Specifically, we formulate training as the dynamic integration of three experience types: Expert Prefixes, Expert Guided, and Self-Exploration. By executing diverse branching exploration around expert"anchors"and employing a mix policy optimization mechanism, we effectively mitigate distribution shifts and resolve optimization conflicts arising from shared prefixes. Our method dynamically adapts the model's knowledge boundaries, effectively balancing exploration diversity with training efficiency.Experimental results demonstrate that E3-TIR achieves a 6 performance improvement over traditional paradigms on tool-use tasks, while requiring less than 10 of the synthetic data. Furthermore, in terms of ROI, a comprehensive metric integrating performance, data cost, and training efficiency we achieve a 1.46x gain compared to baselines. Code is available at https://github.com/yuki-younai/E3-TIR.

Problem

• Tool-Integrated Reasoning
• Large Language Models
• Reinforcement Learning
• Training Paradigm
• Data Efficiency

Innovation

• Tool-Integrated Reasoning
• Experience Exploitation
• Mixed Policy Optimization
• Distribution Shift Mitigation
• Efficient Exploration