On the Adversarial Robustness of Discrete Image Tokenizers

2026-02-20

AI Summary

The vulnerability of discrete image tokenizers to adversarial attacks remains underexplored, posing potential risks to the security of multimodal systems. This work presents the first systematic investigation into their adversarial fragility, introducing an efficient, task-agnostic, and unsupervised adversarial attack method. Furthermore, we propose a label-free adversarial training strategy to enhance tokenizer robustness. Our approach substantially improves resilience against both unsupervised and end-to-end supervised attacks, while demonstrating strong generalization across downstream tasks including image classification, retrieval, and captioning.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Data transparency and provenance

Abstract

Discrete image tokenizers encode visual inputs as sequences of tokens from a finite vocabulary and are gaining popularity in multimodal systems, including encoder-only, encoder-decoder, and decoder-only models. However, unlike CLIP encoders, their vulnerability to adversarial attacks has not been explored. Ours being the first work studying this topic, we first formulate attacks that aim to perturb the features extracted by discrete tokenizers, and thus change the extracted tokens. These attacks are computationally efficient, application-agnostic, and effective across classification, multimodal retrieval, and captioning tasks. Second, to defend against this vulnerability, inspired by recent work on robust CLIP encoders, we fine-tune popular tokenizers with unsupervised adversarial training, keeping all other components frozen. While unsupervised and task-agnostic, our approach significantly improves robustness to both unsupervised and end-to-end supervised attacks and generalizes well to unseen tasks and data. Unlike supervised adversarial training, our approach can leverage unlabeled images, making it more versatile. Overall, our work highlights the critical role of tokenizer robustness in downstream tasks and presents an important step in the development of safe multimodal foundation models.

Problem

• adversarial robustness
• discrete image tokenizers
• multimodal systems
• adversarial attacks
• tokenizer vulnerability

Innovation

• discrete image tokenizers
• adversarial robustness
• unsupervised adversarial training
• multimodal foundation models
• task-agnostic defense