CacheRL:Multi-Turn Tool-Calling Agents via Cached Rollouts and Hybrid Reward

2026-06-12

AI Summary

This work addresses key challenges in multi-turn tool-augmented reasoning for small-scale agents—namely, the difficulty of transferring large model knowledge, high computational costs of real-time tool execution, and sensitivity to cache noise—by introducing an efficient training and inference framework. The proposed approach constructs supervised fine-tuning data using hybrid reasoning traces, enables execution-free replay via a three-tier fuzzy caching mechanism (CacheAgentLoop), and incorporates a cache-level-aware dynamic reward strategy. Integrating iterative supervised fine-tuning, GRPO optimization, token-level masking, and cache-aware reward modeling, the method achieves a validation reward increase from 0.43 to 0.78 on Qwen3-4B-Thinking, attains 92% accuracy in multi-step tool-use sequences—approaching GPT-5’s 94%—and reduces computational cost by approximately two orders of magnitude.

Abstract

We present CacheRL, a system for training small agent foundation models that achieves 92 percent process accuracy on multi-step tool-calling tasks, approaching GPT-5's 94 percent while requiring 100 times less compute. Our approach addresses three challenges in practical agent training: transferring tool-calling knowledge from large models at scale, enabling reinforcement learning without costly live tool execution, and learning robustly from noisy cached environments. CacheRL introduces three key innovations. First, a hybrid thinking trajectory pipeline augments agent trajectories with LLM-generated reasoning traces, producing training examples that teach models not only what tools to call but also why. Second, the CacheAgentLoop eliminates live execution costs through a three-tier fuzzy cache while preserving trajectory fidelity using token-level masking. Third, a cache-tier-aware reward dynamically adjusts answer-quality weights to avoid penalizing models for cache-induced limitations. Through iterative supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO), CacheRL improves Qwen3-4B-Thinking's validation reward from 0.43 to 0.78. On public agentic tool-calling benchmarks, our model achieves competitive performance against frontier models such as GPT-5. Ablation studies show that removing knowledge transfer reduces performance by 41 percent, while cache-aware rewards contribute a 17 percent improvement. Interestingly, reinforcement learning improves training stability but yields limited gains beyond strong supervised fine-tuning, suggesting that data quality and reward design play a more important role than complex optimization methods in building practical small agent models.

Problem

• tool-calling
• multi-turn agents
• cached environments
• knowledge transfer
• reinforcement learning

Innovation

• CacheRL
• tool-calling agents
• cached rollouts
• hybrid reward
• fuzzy cache