ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models

2025-03-01

AI Summary

Existing Tool-Augmented Language Model (TALM) benchmarks predominantly employ simplified single-turn dialogues, failing to assess critical real-world capabilities—such as proactive clarification under information gaps and dynamic API invocation—required for robust tool interaction. Method: We introduce ToolDial, the first TALM-oriented multi-turn dialogue benchmark, comprising 11,111 realistic, multi-domain dialogues (avg. 8.95 turns) sourced from RapidAPI. We propose a novel dialogue modeling framework supporting clarification requests and dynamic API switching, featuring an IO-compatibility–driven API graph generation method to explicitly encode cross-API dependencies. Additionally, we design a fine-grained multi-action state model with 16 action types (e.g., Request, Clarify, Fail-inform), jointly trained with dialogue-history–guided parameter extraction and action prediction. Contribution/Results: Evaluation reveals that state-of-the-art LMs achieve <70% accuracy on key tasks, confirming ToolDial’s rigor; all data and code are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Tool-Augmented Language Models (TALMs) leverage external APIs to answer user queries across various domains. However, existing benchmark datasets for TALM research often feature simplistic dialogues that do not reflect real-world scenarios, such as the need for models to ask clarifying questions or proactively call additional APIs when essential information is missing. To address these limitations, we construct and release ToolDial, a dataset comprising 11,111 multi-turn dialogues, with an average of 8.95 turns per dialogue, based on APIs from RapidAPI. ToolDial has two key characteristics. First, the dialogues incorporate 16 user and system actions (e.g.,"Request","Clarify","Fail inform") to capture the rich dynamics of real-world interactions. Second, we simulate dialogues where the system requests necessary information from the user based on API documentation and seeks additional APIs if the user fails to provide the required information. To facilitate this process, we introduce a method for generating an API graph that represents input and output compatibility between APIs. Using ToolDial, we evaluate a suite of language models on their ability to predict correct actions and extract input parameter values for API calls from the dialogue history. Modern language models achieve accuracy scores below 70%, indicating substantial room for improvement. We release our dataset and code at https://github.com/holi-lab/ToolDial.

Problem

• Addresses simplistic dialogues in TALM benchmark datasets
• Simulates real-world multi-turn dialogues with API interactions
• Evaluates language models on action prediction and API parameter extraction

Innovation

• Constructed ToolDial dataset with 11,111 dialogues
• Simulated dialogues with 16 user-system actions
• Introduced API graph for input-output compatibility