CLIP-Inspector: Model-Level Backdoor Detection for Prompt-Tuned CLIP via OOD Trigger Inversion

2026-04-10

AI Summary

This work addresses the challenge of detecting model-level backdoors in prompt-tuned CLIP models—stealthy attacks that leave the encoder intact yet manipulate out-of-distribution (OOD) data. The authors propose CLIP-Inspector, a novel method that, under white-box access and with only a small set of unlabeled OOD images, efficiently detects and mitigates such backdoors by reverse-engineering class-specific latent triggers. Unlike prior approaches that rely on encoder anomalies or extensive data sanitization, CLIP-Inspector operates with just 1,000 OOD images in a single iteration. Evaluated across 10 datasets and 4 attack types, it achieves a 94% detection accuracy (identifying 47 out of 50 compromised models) and an AUROC of 0.973, substantially outperforming existing baselines.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Organisations with limited data and computational resources increasingly outsource model training to Machine Learning as a Service (MLaaS) providers, who adapt vision-language models (VLMs) such as CLIP to downstream tasks via prompt tuning rather than training from scratch. This semi-honest setting creates a security risk where a malicious provider can follow the prompt-tuning protocol yet implant a backdoor, forcing triggered inputs to be classified into an attacker-chosen class, even for out-of-distribution (OOD) data. Such backdoors leave encoders untouched, making them undetectable to existing methods that focus on encoder corruption. Other data-level methods that sanitize data before training or during inference, also fail to answer the critical question,"Is the delivered model backdoored or not?"To address this model-level verification problem, we introduce CLIP-Inspector (CI), a backdoor detection method designed for prompt-tuned CLIP models. Assuming white-box access to the delivered model and a pool of unlabeled OOD images, CI reconstructs possible triggers for each class to determine if the model exhibits backdoor behaviour or not. Additionally, we demonstrate that using CI's reconstructed trigger for fine-tuning on correctly labeled triggered inputs enables us to re-align the model and reduce backdoor effectiveness. Through extensive experiments across ten datasets and four backdoor attacks, we demonstrate that CI can reconstruct effective triggers in a single epoch using only 1,000 OOD images, achieving a 94% detection accuracy (47/50 models). Compared to adapted trigger-inversion baselines, CI yields a markedly higher AUROC score (0.973 vs 0.495/0.687), thus enabling the vetting and post-hoc repair of prompt-tuned CLIP models to ensure safe deployment.

Problem

• backdoor detection
• prompt-tuned CLIP
• out-of-distribution
• model-level verification
• vision-language models

Innovation

• backdoor detection
• prompt tuning
• trigger inversion
• out-of-distribution (OOD)
• CLIP