QAVA: Query-Agnostic Visual Attack to Large Vision-Language Models

2025-04-15

AI Summary

This paper introduces Query-Agnostic Visual Adversarial Attacks (QAVA), the first query-agnostic adversarial attack against Large Vision-Language Models (LVLMs), designed to generate a single universal adversarial image that induces incorrect answers across arbitrary, unseen questions—without requiring prior knowledge of any specific query. Methodologically, QAVA formulates a gradient-based cross-question objective function, incorporates multi-question joint perturbation constraints, and synthesizes robust perturbations directly in the visual feature space. Unlike conventional query-specific attacks—which require paired image-question inputs—QAVA significantly enhances attack generality and efficiency. Empirical evaluation across multiple state-of-the-art LVLMs shows that QAVA achieves attack success rates comparable to query-specific methods, while a single generated adversarial image generalizes effectively to dozens of unseen questions. This work uncovers, for the first time, a critical yet previously overlooked vulnerability in LVLMs’ visual robustness under open-ended visual question answering. The code is publicly available.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Question Answering

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

In typical multimodal tasks, such as Visual Question Answering (VQA), adversarial attacks targeting a specific image and question can lead large vision-language models (LVLMs) to provide incorrect answers. However, it is common for a single image to be associated with multiple questions, and LVLMs may still answer other questions correctly even for an adversarial image attacked by a specific question. To address this, we introduce the query-agnostic visual attack (QAVA), which aims to create robust adversarial examples that generate incorrect responses to unspecified and unknown questions. Compared to traditional adversarial attacks focused on specific images and questions, QAVA significantly enhances the effectiveness and efficiency of attacks on images when the question is unknown, achieving performance comparable to attacks on known target questions. Our research broadens the scope of visual adversarial attacks on LVLMs in practical settings, uncovering previously overlooked vulnerabilities, particularly in the context of visual adversarial threats. The code is available at https://github.com/btzyd/qava.

Problem

• Attacks LVLMs to give wrong answers for any question
• Creates robust adversarial examples for unknown queries
• Enhances attack effectiveness without specific target questions

Innovation

• Query-agnostic visual attack for LVLMs
• Robust adversarial examples for unknown questions
• Enhances attack effectiveness without question specificity