A Solution-based LLM API-using Methodology for Academic Information Seeking

2024-05-24

arXiv.org

AI Summary

Large language models (LLMs) struggle with multi-API compositional queries in academic information retrieval due to complex, interdependent API invocation logic. Method: This paper proposes a *decomposition-driven code-based reasoning* paradigm and introduces SoAy—a novel framework that models intricate API dependencies as predefined, reusable structured decompositions (i.e., executable API call sequences), substantially reducing LLM reasoning overhead. It further designs a clone-based API sandbox environment—SoAyBench/SoAyEval—for safe, controllable evaluation and training, and deeply integrates the AMiner academic API ecosystem. Contribution/Results: On SoAyBench, SoAy achieves absolute accuracy improvements of 34.58–75.99% over state-of-the-art methods. All data, models, and an online service are fully open-sourced to ensure complete reproducibility.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Applying large language models (LLMs) for academic API usage shows promise in reducing researchers' academic information seeking efforts. However, current LLM API-using methods struggle with complex API coupling commonly encountered in academic queries. To address this, we introduce SoAy, a solution-based LLM API-using methodology for academic information seeking. It uses code with a solution as the reasoning method, where a solution is a pre-constructed API calling sequence. The addition of the solution reduces the difficulty for the model to understand the complex relationships between APIs. Code improves the efficiency of reasoning. To evaluate SoAy, we introduce SoAyBench, an evaluation benchmark accompanied by SoAyEval, built upon a cloned environment of APIs from AMiner. Experimental results demonstrate a 34.58-75.99% performance improvement compared to state-of-the-art LLM API-based baselines. All datasets, codes, tuned models, and deployed online services are publicly accessible at https://github.com/RUCKBReasoning/SoAy.

Problem

• Addresses complex API coupling in academic queries
• Reduces difficulty in understanding API relationships
• Improves reasoning efficiency for information seeking

Innovation

• Solution-based API calling sequence methodology
• Code reasoning improves efficiency
• Pre-constructed API sequences reduce complexity