MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization

2026-07-16

AI Summary

This work addresses the limitations of existing warehouse-scale issue localization research, which often neglects visual evidence and conflates localization with repair tasks in multimodal benchmarks, thereby hindering accurate assessment of visual utility. We propose MM-IssueLoc—the first multimodal benchmark that explicitly treats visual evidence as a distinct variable and decouples localization from repair—providing file- and function-level annotations along with structured visual-textual data generated via Visual Context Extraction (VCE). Through a dual-channel evaluation protocol (with and without images) integrated with multimodal large language models and retrieval systems (e.g., MM-IssueLoc-VL-Emb), our framework enables the first fine-grained analysis of visual information’s contribution to localization. Experiments reveal that state-of-the-art methods achieve only 38.96% Acc@5 at the file level and 22.45% Acc@10 at the function level (with retrievers reaching 33.86%), highlighting substantial room for improvement in multimodal issue localization.

Abstract

Real repository issues routinely include visual evidence such as screenshots, error dialogs, rendered UI states, and logs, yet repository-level issue localization is evaluated mostly as a text-only task. Existing multimodal SE benchmarks evaluate end-to-end repair, entangling localization with patch synthesis and obscuring whether visual input helped, hurt, or was ignored. We introduce \textbf{MM-IssueLoc}, a controlled benchmark and evaluation protocol for repository-level localization with visual evidence. MM-IssueLoc contains 652 issue-PR instances across 23 languages, with annotations for 7 image categories and 4 relevance levels. It provides file-level and function-level gold labels, paired text-only and with-image evaluation, and VCE-based diagnostics that convert images into structured textual evidence. We evaluate LLM-based and retrieval-based systems, including MM-IssueLoc-VL-Emb as a controlled multimodal retriever. Results show that existing systems remain far from reliable multimodal repository localization: the strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10. Cross-benchmark comparisons show that high localization scores on text-dominant SWE benchmarks do not transfer cleanly to multimodal issue localization. MM-IssueLoc turns visual evidence into an explicit evaluation variable, enabling future work to test whether systems improve by using visual evidence for localization, rather than by relying on text-only cues or downstream patch-generation effects.

Problem

• multimodal
• issue localization
• visual evidence
• repository-level
• benchmark

Innovation

• multimodal issue localization
• visual evidence
• controlled benchmark
• repository-level debugging
• VCE-based diagnostics