Hierarchically Robust Zero-shot Vision-language Models

2026-04-20

AI Summary

This work addresses the insufficient robustness of existing zero-shot vision-language models against adversarial attacks targeting both superclass and base-class levels, a challenge exacerbated by conventional fine-tuning methods that often degrade natural accuracy. To overcome this, the authors propose a hierarchical embedding adversarial fine-tuning framework that leverages a multi-level image-text alignment mechanism to regulate the depth of visual embeddings within the class hierarchy. The approach establishes a theoretical connection between embedding depth and the maximal feasible margin, enabling robust representation learning. Furthermore, it supports semantic modeling with multiple tree structures to enhance semantic diversity. Experimental results demonstrate that the method significantly improves robustness against multi-level adversarial attacks across several benchmark datasets while preserving or even enhancing natural accuracy.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Vision-Language Models (VLMs) can perform zero-shot classification but are susceptible to adversarial attacks. While robust fine-tuning improves their robustness, existing approaches align fixed text embeddings with an image embedding, sacrificing natural performance and robustness. A robustness degradation also occurs when a model faces adversarial attacks targeting superclasses (parent classes, e.g., mammal) in addition to their base (leaf) classes (e.g., cat). Thus, to enhance adversarial robustness and leverage the inherent hierarchical properties of class space, we propose a novel adversarial fine-tuning framework based on hierarchical embeddings and several levels of adversarially robust alignment of image-text modalities. Additional mechanisms place visual embeddings at the desired depth of hierarchy, and we provide a theoretical connection between the depth of embedding in the hierarchy and the maximum viable margin size. Our model naturally realizes several margin sizes, boosting generalization of adversaries for robustification. As various trees with different parent labels can share the same leaf labels, we also consider aligning over multiple trees to boost semantic variety. Experiments across several datasets are performed.

Problem

• zero-shot vision-language models
• adversarial robustness
• hierarchical classification
• adversarial attacks
• class hierarchy

Innovation

• hierarchical embeddings
• adversarial robustness
• vision-language models
• zero-shot classification
• multi-tree alignment