🤖 AI Summary
To address incomplete information in natural language user queries, this paper proposes a novel framework integrating large language models (LLMs), Answer Set Programming (ASP), and PDDL-based planning. The method employs ASP to generate a logical intermediate representation that explicitly models missing parameters, while PDDL encodes cross-API data-flow constraints to guide a classical planner in synthesizing semantically consistent API invocation sequences. It introduces the `get_info_api` operator—a novel mechanism for structured, on-demand acquisition of missing information—supporting both single- and multi-objective queries as well as dynamic parameter inference. Evaluated on datasets comprising complete and incomplete queries, the framework achieves over 95% accuracy, substantially outperforming LLM-only baselines. This demonstrates the critical role of symbolic reasoning and formal planning in enhancing LLM robustness, controllability, and semantic fidelity in API composition tasks.
📝 Abstract
Recent availability of Large Language Models (LLMs) has led to the development of numerous LLM-based approaches aimed at providing natural language interfaces for various end-user tasks. These end-user tasks in turn can typically be accomplished by orchestrating a given set of APIs. In practice, natural language task requests (user queries) are often incomplete, i.e., they may not contain all the information required by the APIs. While LLMs excel at natural language processing (NLP) tasks, they frequently hallucinate on missing information or struggle with orchestrating the APIs. The key idea behind our proposed approach is to leverage logical reasoning and classical AI planning along with an LLM for accurately answering user queries including identification and gathering of any missing information in these queries. Our approach uses an LLM and ASP (Answer Set Programming) solver to translate a user query to a representation in Planning Domain Definition Language (PDDL) via an intermediate representation in ASP. We introduce a special API"get_info_api"for gathering missing information. We model all the APIs as PDDL actions in a way that supports dataflow between the APIs. Our approach then uses a classical AI planner to generate an orchestration of API calls (including calls to get_info_api) to answer the user query. Our evaluation results show that our approach significantly outperforms a pure LLM based approach by achieving over 95% success rate in most cases on a dataset containing complete and incomplete single goal and multi-goal queries where the multi-goal queries may or may not require dataflow among the APIs.