SafeFix: Targeted Model Repair via Controlled Image Generation

2025-08-12

AI Summary

Vision recognition models often exhibit systematic biases due to underrepresentation of semantic subpopulations—e.g., rare attribute combinations—in training data. To address this, we propose a failure-attribution-driven generative remediation framework: (1) interpretable localization of model failures on specific semantic subpopulations; (2) controllable synthesis of high-fidelity, failure-relevant samples via conditional text-to-image generation; (3) semantic consistency and distribution alignment filtering using large vision-language models (LVLMs); and (4) targeted fine-tuning of the original model. This approach enables precise, robust data augmentation without introducing spurious correlations or degrading generalization. Experiments demonstrate substantial error reduction on rare subpopulations while preserving overall accuracy, establishing a new paradigm for fine-grained fairness and robustness enhancement in visual recognition.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Deep learning models for visual recognition often exhibit systematic errors due to underrepresented semantic subpopulations. Although existing debugging frameworks can pinpoint these failures by identifying key failure attributes, repairing the model effectively remains difficult. Current solutions often rely on manually designed prompts to generate synthetic training images -- an approach prone to distribution shift and semantic errors. To overcome these challenges, we introduce a model repair module that builds on an interpretable failure attribution pipeline. Our approach uses a conditional text-to-image model to generate semantically faithful and targeted images for failure cases. To preserve the quality and relevance of the generated samples, we further employ a large vision-language model (LVLM) to filter the outputs, enforcing alignment with the original data distribution and maintaining semantic consistency. By retraining vision models with this rare-case-augmented synthetic dataset, we significantly reduce errors associated with rare cases. Our experiments demonstrate that this targeted repair strategy improves model robustness without introducing new bugs. Code is available at https://github.com/oxu2/SafeFix

Problem

• Repairing visual recognition models for underrepresented semantic subpopulations
• Generating semantically faithful images for failure cases
• Reducing errors in rare cases without introducing new bugs

Innovation

• Conditional text-to-image model for targeted image generation
• LVLM filtering for semantic and distribution alignment
• Rare-case-augmented synthetic dataset for model retraining