FailureMem: A Failure-Aware Multimodal Framework for Autonomous Software Repair

2026-03-18

AI Summary

This work addresses key limitations in existing multimodal automated program repair methods—namely, rigid pipelines, insufficient fine-grained visual grounding, and poor reuse of past failure experiences. To overcome these challenges, we propose FailureMem, a novel framework that synergistically integrates a hybrid workflow-agent architecture, region-level active visual perception, and a failure memory bank to enable structured localization and flexible reasoning. By leveraging multimodal fusion, region-level visual grounding, and memory-based modeling of historical failures, FailureMem significantly enhances both exploratory capability and knowledge reuse during the repair process. Evaluated on the SWE-bench Multimodal benchmark, FailureMem achieves a 3.7% higher repair success rate compared to GUIRepair.

Technology Category

• Computer Vision: Multi-modal Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Multimodal Automated Program Repair (MAPR) extends traditional program repair by requiring models to jointly reason over source code, textual issue descriptions, and visual artifacts such as GUI screenshots. While recent LLM-based repair systems have shown promising results, existing approaches face several limitations: rigid workflow pipelines restrict exploration during debugging, visual reasoning is often performed over full-page screenshots without localized grounding, and failed repair attempts are rarely transformed into reusable knowledge. To address these challenges, we propose FailureMem, a multimodal repair framework that integrates three key mechanisms: a hybrid workflow-agent architecture that balances structured localization with flexible reasoning, active perception tools that enable region-level visual grounding, and a Failure Memory Bank that converts past repair attempts into reusable guidance. Experiments on SWE-bench Multimodal demonstrate FailureMem improves the resolved rate over GUIRepair by 3.7%.

Problem

• Multimodal Automated Program Repair
• Failure Awareness
• Visual Grounding
• Debugging Workflow
• Repair Knowledge Reuse

Innovation

• Failure Memory Bank
• multimodal program repair
• region-level visual grounding
• hybrid workflow-agent architecture
• active perception