VisualRepair: Dynamic Tool Calling and Region Focusing for Visual Software Issue Repair

2026-07-15

AI Summary

This work addresses the limited ability of existing automated program repair methods to effectively leverage visual information in multimodal bug reports, particularly when dealing with heterogeneous images and extensive irrelevant regions, which hinders accurate fault localization and patch generation. To overcome this challenge, the paper proposes a novel visual software bug repair framework grounded in multimodal large language models (MLLMs), introducing two key innovations: an image-type-aware dynamic tool-calling chain and a test-time adaptive region-focusing mechanism with scalable attention. These components significantly enhance the model’s comprehension of visual inputs—such as UI screenshots—and improve fault localization precision. Evaluated on the SWE-bench Multimodal benchmark, the approach repairs 196 instances on the test set and 25 on the development set, outperforming the current best baseline by 10 and 11 instances, respectively, thereby demonstrating superior robustness and repair diversity.

Abstract

Automated Program Repair (APR) has witnessed significant progress with the advent of Large Language Models (LLMs). However, as modern software systems increasingly expose rich graphical user interfaces, effectively leveraging visual information from bug screenshots has become essential for understanding bugs and generating accurate fixes in multimodal scenarios. Real-world issue reports frequently contain heterogeneous visual attachments including UI screenshots, IDE snapshots, GIFs, and text-centric images, each with distinct visual patterns and domain-specific semantics that impose substantial perceptual demands on MLLMs. Furthermore, bug screenshots often contain large expanses of uninformative and bug-irrelevant regions, distracting the model's attention and limiting patch diversity. To address these challenges, we propose VisualRepair, an MLLM-based framework for visual software issue repair comprising two core modules: Image Type-aware Tool Calling (ITTC), which classifies input images and dynamically invokes a tailored tool-calling chain for robust visual interpretation, and Dynamic Test-time Region Focusing (DTRF), which grounds multiple bug-related region candidates and refines them via an adaptive zoom-in and zoom-out strategy to improve fault localization and promote diverse patch generation. Extensive experiments on the SWE-bench Multimodal benchmark demonstrate that VisualRepair consistently outperforms state-of-the-art approaches. VisualRepair resolves 196 and 25 instances on the test and dev sets, respectively, surpassing the best baseline by 10 and 11 instances. These results highlight the effectiveness of type-aware visual understanding and region-focused localization for automated visual software issue repair.

Problem

• Automated Program Repair
• Multimodal Software Debugging
• Visual Bug Localization
• Large Multimodal Language Models
• GUI-based Software Issues

Innovation

• Visual Software Repair
• Multimodal LLM
• Tool Calling
• Region Focusing
• Automated Program Repair