TrustCLIP: Learning Private Visual Features via Adversarial Reconstruction

2026-07-05

AI Summary

This work addresses the vulnerability of high-level features in vision and vision-language models to generative inversion attacks, which can reconstruct sensitive input data and lead to privacy leakage. To mitigate this risk, the authors propose TrustCLIP, a novel framework that explicitly incorporates generative reconstruction attacks into the optimization objective. TrustCLIP introduces a feature-conditional generator as an explicit privacy adversary and employs adversarial training to learn a privacy-preserving feature projection. This approach significantly reduces the fidelity of reconstructed images while maintaining competitive performance on downstream tasks. Empirical results demonstrate that TrustCLIP effectively defends against generative inversion attacks across both image classification and multimodal large language model settings, achieving a strong balance between utility preservation and privacy protection.

Abstract

Vision and vision-language models rely on high-level visual representations that are increasingly used across recognition, retrieval, and multimodal reasoning pipelines. However, recent advances in generative modeling have shown that such features can often be inverted, enabling realistic reconstructions of the underlying image and raising significant privacy risks. We revisit this problem through the lens of reconstruction and propose TrustCLIP, a reconstruction-driven framework that treats a feature-conditioned generator as an explicit privacy adversary. TrustCLIP learns a projection between encoder features and downstream modules that is explicitly optimized to degrade the reconstructions produced by generative attackers while retaining the necessary signals for downstream tasks. Unlike prior defenses that rely on discriminative privacy metrics, TrustCLIP directly optimizes against a generative reconstruction attacker, targeting a threat not captured by standard evaluation protocols. We demonstrate its effectiveness in both conventional classification and multimodal large language model pipelines. Across these settings, TrustCLIP consistently reduces the fidelity of generative inversions while maintaining downstream task performance. Project page: https://atnikos.github.io/trustclip/

Problem

• privacy
• visual features
• adversarial reconstruction
• feature inversion
• generative modeling

Innovation

• adversarial reconstruction
• privacy-preserving representation
• feature inversion
• vision-language models
• generative attacker