Beyond Native Success: Auditing Deployment-Interface Exposure of CLIP Backdoors

2026-06-16

AI Summary

Existing backdoor attacks on CLIP are evaluated solely in native tasks, failing to reveal their exposure risks and transferability across diverse deployment interfaces. This work proposes DIFE, the first framework to systematically audit backdoored CLIP models across feature extraction, retrieval, and re-ranking interfaces. Our analysis demonstrates that backdoor exposure hinges on component-level footprints rather than holistic checkpoint inspection and reveals that textual-side poisoning inadequately controls the text encoder. Building on these insights, we introduce BadTextTower—a novel backdoor incorporating unified interface modeling, effective footprint diagnosis, cross-interface comparable evaluation, and targeted injection strategies. Experiments show that success in native tasks does not guarantee controllable cross-interface risks; BadTextTower achieves strong exposure in text-conditioned tasks while preserving a nearly clean purely visual pathway.

Abstract

Contrastive Language-Image Pre-training models are widely reused across downstream interfaces, including feature extraction, retrieval, reranking, and selection. Existing CLIP backdoor, however, usually validate attacks on a small attack-native task, leaving unclear whether the same poisoned checkpoint remains exposed, weakens, or becomes not applicable when reused through other interfaces. We introduce DIFE, a Deployment-Interface Footprint Evaluation framework that audits backdoored CLIP checkpoints across deployment interfaces. DIFE makes various evaluations comparable by specifying each interface's component readout, trigger channel, target event, reference condition, and metric. DIFE also introduces effective-footprint diagnosis to identify the reusable CLIP component or component combination that carries exposure and explains where risk transfers. Auditing reproduced CLIP backdoors with DIFE reveals a structured landscape: native success is not a checkpoint-level risk certificate, exposure follows component footprints, text-side poisoning does not yield textual-encoder control, and some coupled attacks remain mechanism-bound. This audit reveals a import gapin existing CLIP backdoors: a textual encoder that itself becomes a reusable carrier of adversarial behavior. We therefore introduce BadTextTower to fill this gap. BadTextTower produces strong text-conditioned retrieval, reranking, and selection exposure while leaving visual-only reuse nearly clean.

Problem

• CLIP backdoors
• deployment interfaces
• model reuse
• text encoder
• adversarial behavior

Innovation

• CLIP backdoor
• deployment interface
• DIFE framework
• BadTextTower
• text encoder poisoning