Detecting and Fixing API Misuses of Data Science Libraries Using Large Language Models

2025-09-29

AI Summary

This work addresses pervasive API misuse in data science libraries (e.g., scikit-learn, pandas). We propose DSCHECKER, the first approach that jointly models API documentation and dynamic data context via a demand-driven, agent-style LLM architecture. It incorporates adaptive function-calling mechanisms to emulate realistic developer repair workflows. Our method integrates prompt engineering, API metadata extraction, and data-aware static and dynamic analysis, supporting five major data science libraries. Experimental results show that the best-performing model achieves an F1-score of 61.18% for misuse detection and a repair success rate of 51.28%. Under a more realistic agent-mode evaluation—where the LLM autonomously selects tools and iteratively refines repairs—the F1-score and repair rate are 48.65% and 39.47%, respectively. These results represent a significant advance toward practical, automated API misuse detection and correction using LLMs in data science development.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Data science libraries, such as scikit-learn and pandas, specialize in processing and manipulating data. The data-centric nature of these libraries makes the detection of API misuse in them more challenging. This paper introduces DSCHECKER, an LLM-based approach designed for detecting and fixing API misuses of data science libraries. We identify two key pieces of information, API directives and data information, that may be beneficial for API misuse detection and fixing. Using three LLMs and misuses from five data science libraries, we experiment with various prompts. We find that incorporating API directives and data-specific details enhances Dschecker's ability to detect and fix API misuses, with the best-performing model achieving a detection F1-score of 61.18 percent and fixing 51.28 percent of the misuses. Building on these results, we implement Dschecker agent which includes an adaptive function calling mechanism to access information on demand, simulating a real-world setting where information about the misuse is unknown in advance. We find that Dschecker agent achieves 48.65 percent detection F1-score and fixes 39.47 percent of the misuses, demonstrating the promise of LLM-based API misuse detection and fixing in real-world scenarios.

Problem

• Detecting API misuses in data science libraries using LLMs
• Fixing API misuse errors in scikit-learn and pandas
• Enhancing detection accuracy with API directives and data information

Innovation

• DSCHECKER uses LLMs for API misuse detection
• It incorporates API directives and data details
• Implements adaptive function calling for real scenarios