FoCTTA: Low-Memory Continual Test-Time Adaptation with Focus

2025-02-28

AI Summary

To address the high memory overhead and deployment challenges of continual test-time adaptation (CTTA) on resource-constrained IoT devices, this paper proposes a focal representation-layer adaptation mechanism. Instead of updating all batch normalization (BN) layers in full-batch mode, our method employs gradient sensitivity analysis to dynamically identify only the most distribution-shift-sensitive representation layers and performs lightweight parameter fine-tuning exclusively on them. Furthermore, we design a BN-free lightweight adaptation architecture to drastically reduce memory footprint. Evaluated on CIFAR10-C, CIFAR100-C, and ImageNet-C, our approach achieves average accuracy improvements of 4.5%–14.8%, reduces memory consumption by 3×, and maintains consistent gains even under small-batch inference—marking the first CTTA solution that jointly optimizes for both low memory usage and high accuracy on edge devices.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Adversarial Attacks & Robustness
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Continual adaptation to domain shifts at test time (CTTA) is crucial for enhancing the intelligence of deep learning enabled IoT applications. However, prevailing TTA methods, which typically update all batch normalization (BN) layers, exhibit two memory inefficiencies. First, the reliance on BN layers for adaptation necessitates large batch sizes, leading to high memory usage. Second, updating all BN layers requires storing the activations of all BN layers for backpropagation, exacerbating the memory demand. Both factors lead to substantial memory costs, making existing solutions impractical for IoT devices. In this paper, we present FoCTTA, a low-memory CTTA strategy. The key is to automatically identify and adapt a few drift-sensitive representation layers, rather than blindly update all BN layers. The shift from BN to representation layers eliminates the need for large batch sizes. Also, by updating adaptation-critical layers only, FoCTTA avoids storing excessive activations. This focused adaptation approach ensures that FoCTTA is not only memory-efficient but also maintains effective adaptation. Evaluations show that FoCTTA improves the adaptation accuracy over the state-of-the-arts by 4.5%, 4.9%, and 14.8% on CIFAR10-C, CIFAR100-C, and ImageNet-C under the same memory constraints. Across various batch sizes, FoCTTA reduces the memory usage by 3-fold on average, while improving the accuracy by 8.1%, 3.6%, and 0.2%, respectively, on the three datasets.

Problem

• Reduces memory usage in continual test-time adaptation for IoT devices.
• Focuses on adapting drift-sensitive layers instead of all batch normalization layers.
• Improves adaptation accuracy while significantly lowering memory costs.

Innovation

• Focuses on drift-sensitive representation layers
• Eliminates need for large batch sizes
• Reduces memory usage by updating critical layers