TIER: Trajectory-Invariant Execution Rewards for Multi-Step Tool Composition

2026-05-15

AI Summary

Existing reinforcement learning approaches for multi-step tool-composition tasks are hindered by sparse rewards and reliance on reference trajectories, limiting their ability to support diverse solution paths. This work proposes TIER, a novel framework that introduces the first trajectory-invariant dense reward mechanism without requiring reference trajectories. By integrating functional pattern parsing, runtime execution validation, and hierarchical reward modeling, TIER generates fine-grained, interpretable sequence-level supervision signals across four dimensions: format, pattern, execution, and final answer. The method accommodates any valid execution path and adapts robustly to changes in tool interfaces. Evaluated on DepthBench (1–6 steps), TIER achieves over 90% accuracy consistently, significantly outperforming trajectory-supervised baselines, and demonstrates sustained performance gains on BFCL v3 and NestFUL benchmarks.

Abstract

Tool use enables large language models to solve complex tasks through sequences of API calls, yet existing reinforcement learning approaches fail to scale to multi-step composition settings. Outcome-based rewards provide only sparse feedback, while trajectory-supervised rewards depend on annotated reference solutions, penalizing valid alternatives and limiting scalability. We propose TIER: Trajectory-Invariant Execution Rewards, a reward framework that derives supervision directly from function schemas and runtime execution, rather than from reference trajectories. The reward decomposes into format validity, schema adherence, execution success, and answer correctness, providing dense, interpretable sequence-level feedback derived from fine-grained verification of individual steps of tool use. This design allows any valid execution path to receive credit, naturally supporting multiple solution strategies and adapting to evolving tool interfaces. On DepthBench, a compositional benchmark stratified by depth (1 to 6 steps), TIER achieves >90% accuracy across steps, where trajectory-supervised rewards collapse beyond step-4. We further demonstrate consistent gains on benchmarks like BFCL v3 and NestFUL. Ablation studies confirm that all reward components are necessary, highlighting the importance of multi-level supervision for compositional reasoning.

Problem

• multi-step tool composition
• reinforcement learning
• trajectory-supervised rewards
• sparse feedback
• scalability

Innovation

• trajectory-invariant rewards
• tool composition
• execution-based supervision
• multi-step reasoning
• reinforcement learning