Distill Once, Adapt Life-Long: Exploring Dataset Distillation for Continual Test-Time Adaptation

2026-06-18

AI Summary

This work addresses the challenges of error accumulation and catastrophic forgetting in source-data-free continual test-time adaptation (CTTA) under long-term distribution shifts. To mitigate these issues, the authors propose the DO-ALL framework, which first employs data distillation to generate a small set of synthetic anchor points that compactly encode prior knowledge from the source domain. During online adaptation, target samples are matched with semantically closest anchors to jointly enable source replay, representation alignment, and manifold smoothing regularization. Notably, this is the first approach to integrate data distillation into source-free CTTA, reconstructing the source distribution in a compact and privacy-preserving manner. The method significantly enhances long-term robustness and is designed as a plug-and-play module compatible with existing CTTA algorithms. Extensive experiments on CIFAR100-C, ImageNet-C, and the CCC benchmark demonstrate its effectiveness.

Abstract

Continual Test-Time Adaptation (CTTA) aims to maintain model performance under evolving target domains by adapting online without labeled data. However, practical deployments often cannot retain the source dataset due to privacy or licensing constraints, and purely source-free CTTA methods tend to become unstable under long-term distribution shift, suffering from compounding self-training errors and catastrophic forgetting. We introduce DO-ALL (Distill Once, Adapt Life-Long), a plug-and-play framework that revisits source information in a compact and privacy-conscious form via Dataset Distillation (DD). Before deployment, DO-ALL performs DD to produce a small set of synthetic distilled anchors that summarize the source distribution. During adaptation, each target sample is matched with its most semantically aligned anchor, which provides a stable reference for various CTTA via source replay, representation alignment, and manifold-smoothing regularization. DO-ALL can be seamlessly integrated into existing CTTA algorithms, consistently improving long-term robustness across CIFAR100-C, ImageNet-C, and the CCC benchmark. This demonstrates the potential of leveraging DD to enable stable and continuous adaptation without retaining raw source data. The code is available at https://github.com/blue-531/DOALL.

Problem

• Continual Test-Time Adaptation
• Dataset Distillation
• Source-Free Adaptation
• Distribution Shift
• Catastrophic Forgetting

Innovation

• Dataset Distillation
• Continual Test-Time Adaptation
• Source-Free Adaptation
• Synthetic Anchors
• Long-Term Robustness