FOCUS: Frequency-Optimized Conditioning of DiffUSion Models for mitigating catastrophic forgetting during Test-Time Adaptation

2025-08-20

AI Summary

To address catastrophic forgetting of task-relevant knowledge under domain shift in test-time adaptation (TTA), this paper proposes a frequency-optimized input adaptation framework based on diffusion models. The method operates without updating model parameters, preserving original task knowledge while adapting to corrupted inputs. Key contributions include: (i) the first introduction of a frequency-domain conditioning mechanism; (ii) a lightweight Y-shaped Frequency Prediction Network (Y-FPN) that decouples and retains semantically critical low-frequency components; (iii) FrequencyMix for enhanced frequency-domain robustness; and (iv) pseudo-label co-optimization for improved consistency. Extensive experiments across 15 noise types and three benchmark datasets demonstrate state-of-the-art performance on semantic segmentation and monocular depth estimation, with reduced computational overhead compared to existing TTA approaches.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

Test-time adaptation enables models to adapt to evolving domains. However, balancing the tradeoff between preserving knowledge and adapting to domain shifts remains challenging for model adaptation methods, since adapting to domain shifts can induce forgetting of task-relevant knowledge. To address this problem, we propose FOCUS, a novel frequency-based conditioning approach within a diffusion-driven input-adaptation framework. Utilising learned, spatially adaptive frequency priors, our approach conditions the reverse steps during diffusion-driven denoising to preserve task-relevant semantic information for dense prediction. FOCUS leverages a trained, lightweight, Y-shaped Frequency Prediction Network (Y-FPN) that disentangles high and low frequency information from noisy images. This minimizes the computational costs involved in implementing our approach in a diffusion-driven framework. We train Y-FPN with FrequencyMix, a novel data augmentation method that perturbs the images across diverse frequency bands, which improves the robustness of our approach to diverse corruptions. We demonstrate the effectiveness of FOCUS for semantic segmentation and monocular depth estimation across 15 corruption types and three datasets, achieving state-of-the-art averaged performance. In addition to improving standalone performance, FOCUS complements existing model adaptation methods since we can derive pseudo labels from FOCUS-denoised images for additional supervision. Even under limited, intermittent supervision with the pseudo labels derived from the FOCUS denoised images, we show that FOCUS mitigates catastrophic forgetting for recent model adaptation methods.

Problem

• Mitigating catastrophic forgetting during test-time domain adaptation
• Balancing knowledge preservation and domain shift adaptation
• Preserving task-relevant semantic information in diffusion models

Innovation

• Frequency-based conditioning in diffusion models
• Y-shaped Frequency Prediction Network for efficiency
• FrequencyMix data augmentation for robustness