Continuous Expert Assembly: Instance-Conditioned Low-Rank Residuals for All-in-One Image Restoration

2026-05-07

AI Summary

Real-world image degradation exhibits unknown, spatially non-uniform, and composite characteristics, making it challenging for existing all-in-one restoration methods to accurately adapt to local degradation patterns without test-time degradation labels. This work proposes a Continuous Expert Assembly (CEA) framework that dispenses with external prompts, static expert pools, and discrete Top-k selection. Instead, it employs a lightweight cross-attention hyper-adapter to dynamically generate instance-conditioned low-rank bases and residual directions for each spatial token, enabling token-level dense parameterization via signed dot-product affinities. By incorporating a linear attention perspective, the method enhances routing interpretability while achieving significant performance gains over strong baselines on AIO-3, AIO-5, and CDD-11 benchmarks—particularly excelling in scenarios involving spatially varying and composite degradations—all while maintaining low parameter count, computational overhead, and efficient inference.

Abstract

Real-world image degradation is often unknown, spatially non-uniform, and compositional, requiring all-in-one restoration models to adapt a single set of weights to diverse local corruption patterns without test-time degradation labels. Existing methods typically modulate a shared backbone with global prompts or degradation descriptors, or route features through predefined expert pools. However, compact global conditioning can bottleneck localized degradation evidence, while static expert routing may produce homogeneous updates or rely on unstable sparse assignments. We propose \textbf{Continuous Expert Assembly} (CEA), a token-wise dynamic parameterization framework for all-in-one image restoration. CEA employs a lightweight \textbf{Cross-Attention Hyper-Adapter} to probe intermediate spatial features and synthesize instance-conditioned low-rank routing bases and residual directions. Each spatial token then assembles its own residual update via dense signed dot-product affinities over the generated rank-wise components, avoiding external prompts, static expert banks, and discrete Top- selection. The resulting assembly rule also admits a linear-attention perspective, making its dense token-wise routing behavior transparent. Experiments on AIO-3, AIO-5, and CDD-11 show that CEA improves average restoration quality over strong prompt-, descriptor-, and expert-based baselines, with the clearest gains on spatially varying and compositional degradations, while maintaining favorable parameter, FLOP, and runtime efficiency.

Problem

• image restoration
• spatially non-uniform degradation
• compositional degradation
• all-in-one restoration
• degradation adaptation

Innovation

• Continuous Expert Assembly
• Low-Rank Residuals
• Instance-Conditioned Adaptation
• Dynamic Parameterization
• All-in-One Image Restoration