Towards Effective Code-Integrated Reasoning

2025-05-30

AI Summary

This work addresses three core challenges in code-integrated reasoning: imprecise timing of tool invocation, inefficient utilization of execution feedback, and instability in tool-augmented reinforcement learning (RL) training. To this end, we propose an enhanced tool-augmented RL framework that jointly integrates interactive training with a code interpreter, progressive curriculum learning, and an exploration–stability co-optimization strategy—enabling models to dynamically acquire the capability to determine *when* to invoke tools, *how* to generate executable code, and *how* to iteratively refine reasoning. Evaluated on five mainstream mathematical reasoning benchmarks, our method significantly outperforms strong baselines, achieving an average +4.2% absolute accuracy gain while improving inference efficiency by reducing average step count by 18%. Notably, this is the first systematic study to demonstrate that code integration not only extends model capability boundaries but also yields dual gains—enhancing both reasoning quality and computational efficiency—through a closed-loop feedback mechanism.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

In this paper, we investigate code-integrated reasoning, where models generate code when necessary and integrate feedback by executing it through a code interpreter. To acquire this capability, models must learn when and how to use external code tools effectively, which is supported by tool-augmented reinforcement learning (RL) through interactive learning. Despite its benefits, tool-augmented RL can still suffer from potential instability in the learning dynamics. In light of this challenge, we present a systematic approach to improving the training effectiveness and stability of tool-augmented RL for code-integrated reasoning. Specifically, we develop enhanced training strategies that balance exploration and stability, progressively building tool-use capabilities while improving reasoning performance. Through extensive experiments on five mainstream mathematical reasoning benchmarks, our model demonstrates significant performance improvements over multiple competitive baselines. Furthermore, we conduct an in-depth analysis of the mechanism and effect of code-integrated reasoning, revealing several key insights, such as the extension of model's capability boundaries and the simultaneous improvement of reasoning efficiency through code integration. All data and code for reproducing this work are available at: https://github.com/RUCAIBox/CIR.

Problem

• Enhancing code-integrated reasoning with tool-augmented RL
• Improving training stability for effective code tool usage
• Boosting reasoning performance via dynamic code integration

Innovation

• Code-integrated reasoning with interpreter feedback
• Tool-augmented reinforcement learning for stability
• Enhanced training balancing exploration and stability