SVRepair: Structured Visual Reasoning for Automated Program Repair

2026-02-05

AI Summary

This work proposes SVRepair, a multimodal automated program repair framework that leverages structured visual representations to overcome the limitations of existing single-modality approaches, which often fail to exploit critical diagnostic cues from visual artifacts such as screenshots and control flow graphs. SVRepair unifies heterogeneous visual inputs into semantic scene graphs through iterative visual region segmentation and employs a multimodal large language model agent to perform precise fault localization and patch generation. By effectively bridging the gap between visual observations and code semantics, the method significantly mitigates interference from irrelevant context and reduces model hallucination. Experimental results demonstrate that SVRepair achieves state-of-the-art performance, with accuracy rates of 36.47%, 38.02%, and 95.12% on SWE-Bench M, MMCode, and CodeVision benchmarks, respectively.

Technology Category

• Computer Vision: Multi-modal Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large language models (LLMs) have recently shown strong potential for Automated Program Repair (APR), yet most existing approaches remain unimodal and fail to leverage the rich diagnostic signals contained in visual artifacts such as screenshots and control-flow graphs. In practice, many bug reports convey critical information visually (e.g., layout breakage or missing widgets), but directly using such dense visual inputs often causes context loss and noise, making it difficult for MLLMs to ground visual observations into precise fault localization and executable patches. To bridge this semantic gap, we propose \textbf{SVRepair}, a multimodal APR framework with structured visual representation. SVRepair first fine-tunes a vision-language model, \textbf{Structured Visual Representation (SVR)}, to uniformly transform heterogeneous visual artifacts into a \emph{semantic scene graph} that captures GUI elements and their structural relations (e.g., hierarchy), providing normalized, code-relevant context for downstream repair. Building on the graph, SVRepair drives a coding agent to localize faults and synthesize patches, and further introduces an iterative visual-artifact segmentation strategy that progressively narrows the input to bug-centered regions to suppress irrelevant context and reduce hallucinations. Extensive experiments across multiple benchmarks demonstrate state-of-the-art performance: SVRepair achieves \textbf{36.47\%} accuracy on SWE-Bench M, \textbf{38.02\%} on MMCode, and \textbf{95.12\%} on CodeVision, validating the effectiveness of SVRepair for multimodal program repair.

Problem

• Automated Program Repair
• Multimodal Learning
• Visual Artifacts
• Fault Localization
• Large Language Models

Innovation

• multimodal program repair
• structured visual representation
• semantic scene graph
• fault localization
• iterative visual segmentation