GeoDetect: Geometric Adversarial Detection for VLPs

2026-07-16

AI Summary

This work addresses the vulnerability of vision-language pretraining models (VLPs) to adversarial attacks and the inadequacy of existing unimodal detection methods in multimodal settings. It first reveals that the embedding space of VLPs exhibits pronounced anisotropy and demonstrates that adversarial examples geometrically deviate significantly from the manifold of benign data. Building on these insights, the authors propose a detection mechanism grounded in geometric analysis of the embedding space and theoretical derivation, which constructs a scoring function based on the geometric distances from a given sample to randomly sampled reference points. The method achieves consistently effective and robust detection across diverse VLP architectures and attack scenarios, including unimodal, multimodal, and adaptive attacks.

Abstract

Vision-language pre-trained models (VLPs) are widely used in real-world applications. However, they remain vulnerable to adversarial attacks. Although adversarial detection methods have demonstrated success in single-modality settings (either vision or language), their effectiveness and reliability in multimodal models such as VLPs remain largely unexplored. In this work, we study the geometry of VLP embedding spaces and observe structured anisotropy that differs from unimodal vision models. Our theoretical analysis shows that under this anisotropic structure, adversarial attacks increase the expected geometric separation between clean and adversarial examples (AEs). Specifically, we demonstrate that AEs consistently exhibit greater expected distances to randomly sampled points than their clean counterparts, indicating that AEs tend to push representations out of manifold regions. Building on these insights, we propose GeoDetect, which leverages these off-manifold deviations via geometric scores to identify AEs. Through comprehensive evaluations, we show that our approach reliably detects AEs across diverse VLP architectures and threat settings, covering unimodal and multimodal attacks as well as adaptive attacks, thereby providing a robust and practical approach to improving the safety and reliability of these models.

Problem

• vision-language pre-trained models
• adversarial detection
• multimodal models
• adversarial attacks
• embedding space geometry

Innovation

• geometric adversarial detection
• vision-language pre-trained models
• anisotropic embedding space
• off-manifold deviation
• multimodal adversarial robustness