Topological Signatures of Adversaries in Multimodal Alignments

2025-01-29

AI Summary

This work addresses the challenge of robustness evaluation for multimodal image-text alignment models (e.g., CLIP, BLIP) under adversarial attacks. We introduce persistent homology—previously unexplored in multimodal adversarial detection—to quantify topological structural changes induced by adversarial perturbations in the joint embedding space. To this end, we propose two novel topological contrastive losses: a single-scale loss based on total persistence and a multi-scale kernel-weighted loss. Furthermore, we construct the first topological-signature-augmented Maximum Mean Discrepancy (MMD) framework for adversarial detection. Experiments demonstrate that our topological losses exhibit monotonic response behavior across diverse image-text joint attacks, significantly improving detection accuracy and generalization robustness on mainstream benchmarks. Our approach establishes an interpretable, verifiable paradigm for multimodal security, grounded in rigorous topological data analysis.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Multimodal Machine Learning systems, particularly those aligning text and image data like CLIP/BLIP models, have become increasingly prevalent, yet remain susceptible to adversarial attacks. While substantial research has addressed adversarial robustness in unimodal contexts, defense strategies for multimodal systems are underexplored. This work investigates the topological signatures that arise between image and text embeddings and shows how adversarial attacks disrupt their alignment, introducing distinctive signatures. We specifically leverage persistent homology and introduce two novel Topological-Contrastive losses based on Total Persistence and Multi-scale kernel methods to analyze the topological signatures introduced by adversarial perturbations. We observe a pattern of monotonic changes in the proposed topological losses emerging in a wide range of attacks on image-text alignments, as more adversarial samples are introduced in the data. By designing an algorithm to back-propagate these signatures to input samples, we are able to integrate these signatures into Maximum Mean Discrepancy tests, creating a novel class of tests that leverage topological signatures for better adversarial detection.

Problem

• Multimodal Machine Learning
• Adversarial Attacks
• CLIP/BLIP Models

Innovation

• Multimodal Machine Learning
• Adversarial Attacks
• Structural Changes Detection