VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies

2026-08-12

AI Summary

Existing benchmarks struggle to evaluate agents’ integrated capabilities under multi-hop reasoning and strategic constraints in tool usage. To address this gap, this work proposes VAKRA—a unified evaluation benchmark encompassing over 8,000 executable APIs across 62 domains—which uniquely integrates API invocation, multi-source knowledge retrieval, and natural language–specified strategy constraints within a multi-hop reasoning framework. Built upon the ReAct architecture, VAKRA decouples model capabilities from agent design. Experimental results show that even the best-performing models achieve 70.4% accuracy on single-hop tasks, which drops to 50–51% on compositional API tasks and further plummets to 2.4% on unanswered questions under strategy constraints. Performance bottlenecks primarily stem from entity disambiguation and cross-source alignment. The benchmark supports multi-path correctness verification and real API re-execution, establishing a new evaluation paradigm for complex tool-use scenarios.

Abstract

Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities in isolation. We introduce VAKRA (e\textbf{V}aluating \textbf{A}PI and \textbf{K}nowledge \textbf{R}etrieval \textbf{A}gents), a benchmark of over $8{,}000$ executable APIs across $62$ domains with tasks spanning three settings of increasing difficulty: diverse API interaction styles, multi-hop reasoning over structured APIs, and multi-source reasoning with natural-language tool-use policy constraints. Correctness is verified by re-executing predicted tool calls against live APIs, accommodating multiple valid paths. Using a fixed ReAct harness to isolate model capabilities from agent architecture, we evaluate frontier and open-weight models and find that even the best model achieves only 70.4\% on single-hop endpoint-style tasks and drops to 50--51\% on compositional APIs; performance degrades by over 50\% as reasoning depth increases, and policy-constrained questions expose severe failures (as low as 2.4\% on unanswerable queries). Trace analysis shows failures concentrate at language-mediated reasoning - entity disambiguation, cross-source grounding, rather than tool invocation mechanics. Code is available https://github.com/IBM/VAKRA. Dataset is available https://huggingface.co/datasets/ibm-research/VAKRA

Problem

• multi-hop reasoning
• API interaction
• retrieval
• tool-use policies
• enterprise agents

Innovation

• multi-hop reasoning
• tool-use policies
• API retrieval
• agent evaluation benchmark
• compositional reasoning