Rethinking Brain Decoding with CLIP: The Role of Adversarial Robustness

2026-07-03

AI Summary

Standard CLIP models struggle to align effectively with fMRI signals due to the presence of non-robust visual features, which limits brain decoding performance. This work systematically demonstrates for the first time the positive impact of adversarial robustness on neural decoding and proposes using an adversarially trained CLIP variant as the target visual representation to enhance alignment between neural and visual embeddings under a fixed fMRI decoder. Evaluated through image retrieval, zero-shot classification, and attribution analysis on the NSD and GOD datasets, the approach significantly outperforms baseline methods, indicating that robust training yields visual features that better reflect the structure of human perceptual representations.

Abstract

Brain decoding aims to uncover neural mechanisms by inferring stimulus-related representations from brain signals. In fMRI studies, this is typically achieved by mapping fMRI responses to the latent representations of computational models. Recently, CLIP has become a popular choice for brain decoding due to its rich vision--language embedding space. However, aligning fMRI signals with CLIP representations remains challenging. As CLIP is not explicitly optimized for neural alignment, its representations may capture statistically predictive cues that are only partially reflected in brain activity, limiting decoding performance. In this paper, we investigate whether adversarially robust representations improve neural decoding with CLIP. Adversarial training suppresses non-robust features and promotes more stable, perceptually structured representations, which may better align with brain activity. We evaluate this by fixing the fMRI decoder and varying only the target representation (standard CLIP vs. robust variants) on fMRI-image retrieval and zero-shot classification tasks across NSD and GOD datasets. Empirical results show that this simple change consistently improves task performance and yields stronger alignment across multiple metrics. Attribution analysis further reveals consistently low agreement between standard CLIP and its robust variants, suggesting that adversarial robustness reorganizes feature importance in the visual representation. These findings suggest that the choice of target representation influences neural decoding performance and that adversarial robustness may serve as a useful criterion for brain decoding.

Problem

• brain decoding
• CLIP
• adversarial robustness
• fMRI
• neural alignment

Innovation

• adversarial robustness
• brain decoding
• CLIP
• fMRI
• neural alignment