Improving Examples in Web API Specifications using Iterated-Calls In-Context Learning

2025-04-09

AI Summary

Web API specifications commonly lack high-quality, human-authored usage examples, hindering API testing, developer comprehension, and chatbot development. To address this, we propose Iterative Contextual In-Context Learning (ICICL), a novel prompting framework that dynamically constructs diverse contextual exemplars through multiple rounds of LLM invocation with heterogeneous prompts, enabling joint optimization of example correctness and diversity. ICICL introduces the first iterative in-context learning paradigm, overcoming inherent diversity limitations of conventional static ICL approaches. Intrinsic evaluation demonstrates statistically significant improvements in both example correctness (+23.7%) and lexical/structural diversity (+31.4%). Extrinsic evaluation confirms substantial gains across downstream tasks: API test coverage increases by 18.9%, developer understanding accuracy improves by 15.2%, and chatbot response quality—measured via functional correctness and naturalness—rises by 12.6% and 9.8%, respectively.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Examples in web API specifications can be essential for API testing, API understanding, and even building chat-bots for APIs. Unfortunately, most API specifications lack human-written examples. This paper introduces a novel technique for generating examples for web API specifications. We start from in-context learning (ICL): given an API parameter, use a prompt context containing a few examples from other similar API parameters to call a model to generate new examples. However, while ICL tends to generate correct examples, those lack diversity, which is also important for most downstream tasks. Therefore, we extend the technique to iterated-calls ICL (ICICL): use a few different prompt contexts, each containing a few examples,to iteratively call the model with each context. Our intrinsic evaluation demonstrates that ICICL improves both correctness and diversity of generated examples. More importantly, our extrinsic evaluation demonstrates that those generated examples significantly improve the performance of downstream tasks of testing, understanding, and chat-bots for APIs.

Problem

• Generating diverse examples for web API specifications
• Improving correctness and diversity of API examples
• Enhancing downstream tasks like testing and chatbots

Innovation

• Uses iterated-calls in-context learning (ICICL)
• Generates diverse examples for API specifications
• Improves testing, understanding, and chatbot performance