CallNavi: A Study and Challenge on Function Calling Routing and Invocation in Large Language Models

2025-01-09

AI Summary

This work addresses the challenge of enabling large language models (LLMs) to accurately identify, route, and invoke nested API functions in complex, multi-step tasks. To this end, we propose three key contributions: (1) a novel benchmark dataset—the first to comprehensively cover the full API invocation pipeline, including discovery, selection, parameter generation, and nested execution; (2) a hybrid routing framework that decouples function selection (performed by a general-purpose LLM) from parameter generation (handled by a lightweight fine-tuned model); and (3) structured prompt engineering specifically designed for nested API calls to improve robustness. Experimental results demonstrate that our approach achieves an average 32.7% improvement in parameter correctness on multi-step nested API tasks, significantly enhancing generalization capability and practical deployability. The method provides a scalable, modular foundation for building API-driven conversational systems.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Interacting with a software system via a chatbot can be challenging, especially when the chatbot needs to generate API calls, in the right order and with the right parameters, to communicate with the system. API calling in chatbot systems poses significant challenges, particularly in complex, multi-step tasks requiring accurate API selection and execution. We contribute to this domain in three ways: first, by introducing a novel dataset designed to assess models on API function selection, parameter generation, and nested API calls; second, by benchmarking state-of-the-art language models across varying levels of complexity to evaluate their performance in API function generation and parameter accuracy; and third, by proposing an enhanced API routing method that combines general-purpose large language models for API selection with fine-tuned models for parameter generation and some prompt engineering approach. These approaches lead to substantial improvements in handling complex API tasks, offering practical advancements for real-world API-driven chatbot systems.

Problem

• Large Language Models
• Function Calling
• Complex Task Handling

Innovation

• CallNavi Method
• Enhanced Function Calling
• Integrated Model Approach