VisDocSketcher: Towards Scalable Visual Documentation with Agentic Systems

2025-09-15

AI Summary

This study addresses the subjectivity and lack of standardization in quality assessment for automated code visualization documentation. We propose the first end-to-end visual documentation generation and evaluation framework powered by LLM agents. Methodologically, it integrates static code analysis with multi-stage LLM agent collaboration: prompt engineering and structural code parsing identify key logical units, enabling scalable, context-aware diagram generation. Crucially, we introduce AutoSketchEval—a novel automated evaluation framework incorporating objective, code-level metrics (e.g., coverage, consistency, executability) to quantify documentation quality reliably. Experiments demonstrate that our approach generates effective visual documentation for 74.4% of test cases—outperforming template-based baselines by +26.7%–39.8%. Furthermore, AutoSketchEval achieves an AUC ≥ 0.87 in predicting human-validated quality, confirming simultaneous advances in both generation fidelity and evaluation rigor.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Evaluation and Analysis
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Visual documentation is an effective tool for reducing the cognitive barrier developers face when understanding unfamiliar code, enabling more intuitive comprehension. Compared to textual documentation, it provides a higher-level understanding of the system structure and data flow. Developers usually prefer visual representations over lengthy textual descriptions for large software systems. Visual documentation is both difficult to produce and challenging to evaluate. Manually creating it is time-consuming, and currently, no existing approach can automatically generate high-level visual documentation directly from code. Its evaluation is often subjective, making it difficult to standardize and automate. To address these challenges, this paper presents the first exploration of using agentic LLM systems to automatically generate visual documentation. We introduce VisDocSketcher, the first agent-based approach that combines static analysis with LLM agents to identify key elements in the code and produce corresponding visual representations. We propose a novel evaluation framework, AutoSketchEval, for assessing the quality of generated visual documentation using code-level metrics. The experimental results show that our approach can valid visual documentation for 74.4% of the samples. It shows an improvement of 26.7-39.8% over a simple template-based baseline. Our evaluation framework can reliably distinguish high-quality (code-aligned) visual documentation from low-quality (non-aligned) ones, achieving an AUC exceeding 0.87. Our work lays the foundation for future research on automated visual documentation by introducing practical tools that not only generate valid visual representations but also reliably assess their quality.

Problem

• Automating visual documentation generation from code
• Overcoming subjective evaluation of visual documentation quality
• Enhancing scalability of visual documentation for large systems

Innovation

• Agentic LLM systems for visual documentation
• Combines static analysis with LLM agents
• Novel evaluation framework using code-level metrics