🤖 AI Summary
Existing benchmarks evaluate single tools in isolation, overlooking real-world challenges such as functional overlap and cross-server orchestration—leading to distorted evaluations. This paper introduces MSC-Bench, the first end-to-end, multi-hop tool-coordination benchmark designed for the hierarchical Model-Context Protocol (MCP) ecosystem. Its contributions are threefold: (1) It constructs reproducible ground truth via “functionally equivalent sets,” enabling objective metrics (e.g., F1-score) and reducing reliance on LLM-based evaluation; (2) It proposes a five-level curriculum-style difficulty taxonomy to systematically diagnose failures in cross-server planning and out-of-scope request handling; (3) It comprehensively covers capabilities from single-tool invocation to complex, multi-server coordination. Experiments expose critical performance bottlenecks of current LLM agents under rigid hierarchical constraints. MSC-Bench is open-sourced, accompanied by a fine-grained attribution analysis framework to support rigorous agent evaluation and improvement.
📝 Abstract
We introduce MSC-Bench, a large-scale benchmark for evaluating multi-hop, end-to-end tool orchestration by LLM agents in a hierarchical Model-Context Protocol (MCP) ecosystem. Existing benchmarks often evaluate tools in isolation, ignoring challenges such as functional overlap and cross-server orchestration, leading to overly optimistic assessments. MSC-Bench addresses these gaps by constructing ground truth through 'equal function sets', allowing objective metrics such as F1 score and reducing the dependency on LLM-as-a-judge evaluation. Organized as a five-level curriculum, it systematically tests agent capabilities from single-tool orchestration to complex cross-server planning, and robustness to out-of-scope requests. Experiments reveal that rigid hierarchies can hinder performance without co-designed strategies, and even state-of-the-art agents exhibit systemic weaknesses in robustness. MSC-Bench provides a diagnostic framework to expose these limitations and guide the development of more capable and efficient tool-using agents. The benchmark and resources are publicly available at https://github.com/snooow1029/MSC_Bench.