Panda: Test-Time Adaptation with Negative Data Augmentation

2025-11-13

AI Summary

Pretrained vision-language models (VLMs) achieve strong zero-shot classification performance but suffer significant robustness degradation under image corruptions. Existing test-time adaptation (TTA) methods rely on positive data augmentation (PDA), which reduces prediction variance yet incurs high computational overhead and fails to mitigate class prediction bias induced by distribution shifts. To address this, we propose Panda—a novel TTA paradigm based on negative data augmentation. Panda generates semantically meaningless negative samples via image patching and non-semantic random recombination, thereby eliminating spurious feature biases; it further suppresses corruption-correlated components via feature mean subtraction. Crucially, Panda enables cross-sample batch-shared augmentation, substantially reducing computational cost. Experiments demonstrate that Panda is compatible with mainstream TTA frameworks and consistently improves zero-shot classification robustness across diverse corruption types, outperforming state-of-the-art PDA-based approaches.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Pretrained VLMs exhibit strong zero-shot classification capabilities, but their predictions degrade significantly under common image corruptions. To improve robustness, many test-time adaptation (TTA) methods adopt positive data augmentation (PDA), which generates multiple views of each test sample to reduce prediction variance. However, these methods suffer from two key limitations. First, it introduces considerable computational overhead due to the large number of augmentations required per image. Second, it fails to mitigate prediction bias, where the model tends to predict certain classes disproportionately under corruption, as PDA operates on corrupted inputs and typically does not remove the corruption itself. To address these challenges, we propose Panda, a novel TTA method based on negative data augmentation (NDA). Unlike positive augmentations that preserve object semantics, Panda generates negative augmentations by disrupting semantic content. It divides images into patches and randomly assembles them from a shared patch pool. These negatively augmented images retain corruption-specific features while discarding object-relevant signals. We then subtract the mean feature of these negative samples from the original image feature, effectively suppressing corruption-related components while preserving class-relevant information. This mitigates prediction bias under distribution shifts. Panda allows augmentation to be shared across samples within a batch, resulting in minimal computational overhead. Panda can be seamlessly integrated into existing test-time adaptation frameworks and substantially improve their robustness. Our experiments indicate that Panda delivers superior performance compared to PDA methods, and a wide range of TTA methods exhibit significantly enhanced performance when integrated with Panda. Our code is available at https://github.com/ruxideng/Panda .

Problem

• Improving robustness of pretrained vision-language models under image corruptions
• Reducing computational overhead of test-time adaptation methods
• Mitigating prediction bias caused by distribution shifts during testing

Innovation

• Uses negative data augmentation for test-time adaptation
• Generates semantic-disrupting patches from shared pool
• Subtracts negative features to suppress corruption bias