Harnessing LLMs for Document-Guided Fuzzing of Python Libraries

2026-08-12

AI Summary

This work addresses the challenge of testing Python library APIs, which often involve complex parameter constraints and cross-parameter dependencies that existing methods struggle to model effectively. The authors propose VistaFuzz, a novel approach that leverages locally deployed open-source large language models to automatically extract parameter specifications and dependency relationships directly from API documentation. These extracted constraints are then integrated into the fuzzing input generation process, enabling documentation-guided, high-efficiency testing. By combining natural language parsing, constraint reasoning, and fuzz testing, VistaFuzz achieves over 95% effective test case generation across 7,718 APIs from 12 widely used Python libraries. The evaluation uncovered 74 distinct issues, 43 of which were confirmed by developers and 29 subsequently fixed.

Abstract

Python libraries underpin deep learning, scientific computing, data analysis, and computer vision, making their reliability critical to downstream applications. Testing their APIs requires inputs that satisfy both per-parameter constraints and dependencies among parameters. Existing approaches either leave such constraints implicit in generated programs or rely on library-specific parsing rules. This paper introduces VistaFuzz, a document-guided fuzzing technique that uses a locally served open-sourced LLM to extract parameter specifications from API documents and generate inputs that satisfy both parameter constraints and inter-parameter dependencies. We evaluate VistaFuzz on 7,718 APIs across twelve Python libraries. Inter-parameter relationships occur in 40.1\% of tested APIs, and disabling their resolution reduces the valid generation rate on those APIs from above 95\% to 31.6\%--52.8\%. VistaFuzz reports 74 issues, of which 43 have been confirmed by developers and 29 have been fixed.

Problem

• API testing
• parameter constraints
• inter-parameter dependencies
• fuzzing
• Python libraries

Innovation

• LLM-guided fuzzing
• API documentation parsing
• parameter dependency resolution
• Python library testing
• document-guided input generation