On the Adversarial Risk of Test Time Adaptation: An Investigation into Realistic Test-Time Data Poisoning

2024-10-07

arXiv.org

AI Summary

This work exposes the inherent vulnerability of test-time adaptation (TTA) to realistic data poisoning attacks. Addressing practical constraints—such as gray-box access and no reliance on clean samples—we propose the first TTA-oriented *targeted* poisoning framework tailored to real-world deployment. We introduce TTA-aware attack objectives and an in-distribution loss function, thereby overcoming classical white-box or strong-assumption limitations. Leveraging TTA-aware gradient-based poisoning, comprehensive robustness benchmarking, and defense analysis, we demonstrate that the robustness of mainstream TTA methods is systematically overestimated under realistic poisoning. Empirical evaluation reveals that while existing TTA methods retain partial resilience against gray-box poisoning, they suffer significant performance degradation. To foster secure TTA research, we release an open-source, reproducible codebase—providing the first formal theoretical modeling, standardized evaluation benchmark, and actionable defense insights for TTA under poisoning threats.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Test-time adaptation (TTA) updates the model weights during the inference stage using testing data to enhance generalization. However, this practice exposes TTA to adversarial risks. Existing studies have shown that when TTA is updated with crafted adversarial test samples, also known as test-time poisoned data, the performance on benign samples can deteriorate. Nonetheless, the perceived adversarial risk may be overstated if the poisoned data is generated under overly strong assumptions. In this work, we first review realistic assumptions for test-time data poisoning, including white-box versus grey-box attacks, access to benign data, attack order, and more. We then propose an effective and realistic attack method that better produces poisoned samples without access to benign samples, and derive an effective in-distribution attack objective. We also design two TTA-aware attack objectives. Our benchmarks of existing attack methods reveal that the TTA methods are more robust than previously believed. In addition, we analyze effective defense strategies to help develop adversarially robust TTA methods. The source code is available at https://github.com/Gorilla-Lab-SCUT/RTTDP.

Problem

• Investigates adversarial risks in test-time adaptation (TTA) methods.
• Proposes realistic test-time data poisoning attack methods without benign data access.
• Analyzes defense strategies to enhance TTA robustness against adversarial attacks.

Innovation

• Proposes realistic test-time data poisoning attack method
• Designs TTA-aware attack objectives for robustness
• Analyzes defense strategies for adversarially robust TTA