BoTTA: Benchmarking on-device Test Time Adaptation

2025-04-14

AI Summary

Existing test-time adaptation (TTA) research lacks systematic investigation under resource constraints typical of mobile/edge devices. Method: This work introduces the first TTA benchmark tailored for edge-device scenarios, addressing practical challenges including few-shot learning, limited categories, and concurrent inter- and intra-sample distribution shifts. We propose a four-dimensional constrained evaluation framework, formalize and empirically assess periodic adaptation paradigms, and conduct system-level measurements (memory footprint, latency, accuracy) on real-world platforms (e.g., Raspberry Pi). We integrate mainstream algorithms—including SHOT—for quantitative analysis of performance–overhead trade-offs under strict resource limits. Contribution/Results: Empirical results reveal severe generalization degradation of current TTA methods under few-shot and unseen-category settings; SHOT incurs up to 1.08× higher peak memory versus baseline. This work fills a critical gap in edge-oriented TTA benchmarking and provides empirically grounded design principles and deployment guidelines for lightweight TTA.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Adversarial Attacks & Robustness
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

The performance of deep learning models depends heavily on test samples at runtime, and shifts from the training data distribution can significantly reduce accuracy. Test-time adaptation (TTA) addresses this by adapting models during inference without requiring labeled test data or access to the original training set. While research has explored TTA from various perspectives like algorithmic complexity, data and class distribution shifts, model architectures, and offline versus continuous learning, constraints specific to mobile and edge devices remain underexplored. We propose BoTTA, a benchmark designed to evaluate TTA methods under practical constraints on mobile and edge devices. Our evaluation targets four key challenges caused by limited resources and usage conditions: (i) limited test samples, (ii) limited exposure to categories, (iii) diverse distribution shifts, and (iv) overlapping shifts within a sample. We assess state-of-the-art TTA methods under these scenarios using benchmark datasets and report system-level metrics on a real testbed. Furthermore, unlike prior work, we align with on-device requirements by advocating periodic adaptation instead of continuous inference-time adaptation. Experiments reveal key insights: many recent TTA algorithms struggle with small datasets, fail to generalize to unseen categories, and depend on the diversity and complexity of distribution shifts. BoTTA also reports device-specific resource use. For example, while SHOT improves accuracy by $2.25 imes$ with $512$ adaptation samples, it uses $1.08 imes$ peak memory on Raspberry Pi versus the base model. BoTTA offers actionable guidance for TTA in real-world, resource-constrained deployments.

Problem

• Evaluating TTA methods for mobile and edge devices
• Addressing limited resources and diverse distribution shifts
• Assessing device-specific resource use and adaptation efficiency

Innovation

• Benchmark for on-device test time adaptation
• Evaluates TTA under mobile resource constraints
• Advocates periodic adaptation over continuous