Improving Adversarial Robustness via Phase and Amplitude-aware Prompting

2025-02-06

AI Summary

To address the insufficient robustness of deep neural networks against adversarial attacks, this paper proposes a phase-and-magnitude-aware prompt defense. It introduces Fourier-domain phase and magnitude spectra into prompt learning—marking the first such integration—and constructs semantic-sensitive, class-adaptive frequency-domain prompts for each category. A robustness-driven dynamic weighting mechanism is designed to select and fuse prompts conditioned on predicted labels during inference, thereby enhancing input robustness. The method synergizes frequency-domain analysis, class-conditional prompt selection, and lightweight adversarial optimization, requiring no retraining of the backbone model. Evaluated on CIFAR-10/100 and ImageNet-C benchmarks under PGD-10 attacks, it achieves over 8% accuracy improvement while preserving high clean accuracy and incurring minimal computational overhead—significantly outperforming existing prompt-based defense approaches.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Measurement, analysis, and circumvention of Web censorship

Abstract

Deep neural networks are found to be vulnerable to adversarial noises. The prompt-based defense has been increasingly studied due to its high efficiency. However, existing prompt-based defenses mainly exploited mixed prompt patterns, where critical patterns closely related to object semantics lack sufficient focus. The phase and amplitude spectra have been proven to be highly related to specific semantic patterns and crucial for robustness. To this end, in this paper, we propose a Phase and Amplitude-aware Prompting (PAP) defense. Specifically, we construct phase-level and amplitude-level prompts for each class, and adjust weights for prompting according to the model's robust performance under these prompts during training. During testing, we select prompts for each image using its predicted label to obtain the prompted image, which is inputted to the model to get the final prediction. Experimental results demonstrate the effectiveness of our method.

Problem

• Enhances adversarial robustness via phase and amplitude-aware prompting.
• Addresses vulnerability of deep neural networks to adversarial noises.
• Focuses on semantic-related patterns for improved defense mechanisms.

Innovation

• Phase and Amplitude-aware Prompting
• Class-specific phase and amplitude prompts
• Weight adjustment based on robust performance