BIR-Adapter: A Low-Complexity Diffusion Model Adapter for Blind Image Restoration

2025-09-08

AI Summary

This work addresses key challenges in blind image restoration—including reliance on auxiliary feature extractors, high computational overhead, and severe hallucination artifacts—by proposing BIR-Adapter, a lightweight diffusion model adapter. Unlike prior approaches, BIR-Adapter requires no auxiliary network training; instead, it directly leverages the self-attention mechanisms of a frozen pre-trained diffusion model to extract degradation-aware features from corrupted inputs, and augments its representational capacity via low-rank adaptation. To suppress sampling-induced artifacts, we introduce a novel sampling guidance strategy, enabling super-resolution-only diffusion models to generalize effectively to unseen degradation types. Extensive experiments on both synthetic and real-world blind restoration benchmarks demonstrate that BIR-Adapter achieves or surpasses state-of-the-art performance, with ~70% fewer parameters and 2.3× faster inference. The method thus offers superior efficiency, generalizability across degradation types, and strong cross-task transferability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Adversarial Learning & Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

This paper introduces BIR-Adapter, a low-complexity blind image restoration adapter for diffusion models. The BIR-Adapter enables the utilization of the prior of pre-trained large-scale diffusion models on blind image restoration without training any auxiliary feature extractor. We take advantage of the robustness of pretrained models. We extract features from degraded images via the model itself and extend the self-attention mechanism with these degraded features. We introduce a sampling guidance mechanism to reduce hallucinations. We perform experiments on synthetic and real-world degradations and demonstrate that BIR-Adapter achieves competitive or better performance compared to state-of-the-art methods while having significantly lower complexity. Additionally, its adapter-based design enables integration into other diffusion models, enabling broader applications in image restoration tasks. We showcase this by extending a super-resolution-only model to perform better under additional unknown degradations.

Problem

• Blind image restoration without auxiliary feature extractors
• Reducing hallucinations in diffusion model outputs
• Extending super-resolution models to handle unknown degradations

Innovation

• Low-complexity adapter for diffusion models
• Extends self-attention with degraded image features
• Sampling guidance mechanism reduces hallucinations