NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls

2024-09-04

arXiv.org

AI Summary

This work addresses the limited planning and execution capabilities of large language models (LLMs) in complex, deeply nested API call sequences. To this end, we introduce NESTFUL—the first executable benchmark specifically designed for multi-level nested function chains, comprising over 1,800 real-world, API-driven input-output dependencies (i.e., prior-output → subsequent-input links). We propose a novel, structured, multi-tiered evaluation paradigm that quantifies LLMs’ bottlenecks in nested reasoning for the first time, integrating functional correctness verification, full-sequence exact match, and pairwise win-rate comparison as complementary metrics. Experimental results reveal that state-of-the-art models achieve only 25% full-sequence accuracy and a 34% win rate, exposing fundamental limitations in deep tool orchestration. NESTFUL thus provides a reproducible, scalable evaluation framework and concrete guidance for advancing tool-augmented LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The resurgence of autonomous agents built using large language models (LLMs) to solve complex real-world tasks has brought increased focus on LLMs' fundamental ability of tool or function calling. At the core of these agents, an LLM must plan, execute, and respond using external tools, APIs, and custom functions. Research on tool calling has gathered momentum, but evaluation benchmarks and datasets representing the complexity of the tasks have lagged behind. In this work, we focus on one such complexity, nested sequencing, with the goal of extending existing benchmarks and evaluation. Specifically, we present NESTFUL, a benchmark to evaluate LLMs on nested sequences of API calls, i.e., sequences where the output of one API call is passed as input to a subsequent call. NESTFUL contains 1800+ nested sequences where all the function calls are executable. Experimental results on multiple models and settings show that the best-performing model on the dataset has a full sequence match accuracy of 25% and win-rate of 34% necessitating a large scope for improvement in the nested sequencing aspect of function calling. Our analysis of these results provides possible future research directions for the community, in addition to a benchmark to track progress. We have released the NESTFUL dataset under the Apache 2.0 license at https://github.com/IBM/NESTFUL.

Problem

• Large Language Models
• Complex API Calls
• Tool Utilization

Innovation

• NESTFUL
• Nested API Calls
• Language Model Evaluation