When Retrieval Metrics Mislead: Measuring Policy Signal in Long-Horizon Tool-Use Agents

2026-06-22

AI Summary

This study addresses the limitation of traditional retrieval metrics—such as exact-match recall—in capturing the practical utility of retrieved content for downstream decision-making models. Using the tau-bench benchmark, the authors evaluate retrieval strategies by fine-tuning Qwen2.5-3B/7B classifiers to perform pre-action classification within a closed-loop classification framework. Through structured state representations, multiple controlled baselines (gold policy, mismatched policy, and no policy), and confidence interval analysis, they find that although only 7% of retrieved results rank the correct policy first, the resulting macro-F1 score reaches 0.58—close to the gold policy’s 0.60 and substantially outperforming baseline conditions (0.32 and 0.21). These findings indicate that conventional recall metrics significantly underestimate the real-world effectiveness of retrieval strategies in long-horizon tool-use scenarios.

Abstract

Exact-match retrieval recall is often used as a proxy for whether a retriever supplies useful policy context to a downstream decision model. We test this proxy for pre-action policy classification in tau-bench using Qwen2.5-3B/7B classifiers. Under gold-policy conditioning, a compact structured state improves macro-F1 over raw trajectories by 0.13-0.17 after tuning. We then replace the benchmark-designated policy clause with the top-ranked clause retrieved from decision-time context. Although the exact governing clause is retrieved at rank 1 for only 7% of airline states, the primary 3B classifier obtains macro-F1 0.58 with retrieved clauses versus 0.60 with gold clauses (Delta=-0.02, task-cluster 95% CI [-0.23,+0.21]); mismatched-policy and no-policy controls score 0.32 and 0.21. We do not detect a macro-F1 difference between retrieved and gold clauses in this configuration, although the interval remains too wide to establish non-inferiority. The same qualitative pattern appears with a second retriever and at 7B, while varying across fine-tuning configurations. These results indicate that exact-match clause recall can underestimate downstream policy utility in this benchmark setting, motivating evaluation with retrieved policies in the classification loop rather than recall alone.

Problem

• retrieval metrics
• policy signal
• long-horizon tool-use
• exact-match recall
• downstream utility

Innovation

• retrieval metrics
• policy signal
• tool-use agents
• exact-match recall
• downstream utility