ADC: Enhancing Function Calling Via Adversarial Datasets and Code Line-Level Feedback

2024-12-23

AI Summary

Large language models (LLMs) exhibit persistent format adherence deviations and inaccurate multi-parameter matching in complex function calls, undermining robustness and accuracy. To address this, we propose a collaborative optimization framework comprising three core innovations: (1) a novel line-level execution feedback supervision mechanism for fine-grained error localization; (2) adversarial data generation to enhance coverage of edge-case scenarios; and (3) a phased incremental training paradigm that progressively refines function understanding and invocation capabilities. Our approach is built upon a code-finetuning dataset enriched with execution trace feedback and structured adversarial samples. Evaluated on the BFCL benchmark, our method achieves significant gains in both accuracy and generalization—marking the first instance of stable function invocation across multi-hop, deeply nested, and high-dimensional parameter settings. This establishes a new state-of-the-art for complex function interface comprehension.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Language Models (LLMs) have made significant strides in Natural Language Processing and coding, yet they struggle with robustness and accuracy in complex function calls. To tackle these challenges, this paper introduces ADC, an innovative approach that enhances LLMs' ability to follow function formats and match complex parameters. ADC utilizes a high-quality code fine-tuning dataset with line-level execution feedback, providing granular process supervision that fosters strong logical reasoning and adherence to function formats. It also employs an adversarial dataset generation process to improve parameter matching. The staged training methodology capitalizes on both enriched code datasets and refined adversarial datasets, leading to marked improvements in function calling capabilities on the Berkeley Function-Calling Leaderboard (BFCL) Benchmark. The innovation of ADC lies in its strategic combination of process supervision, adversarial refinement, and incremental learning, setting a new standard for LLM proficiency in complex function calling.

Problem

• Large Language Models
• Complex Function Calling
• Accuracy Improvement

Innovation

• Advanced Code Data
• Puzzle Training
• Supervised Learning