Gaslight, Gatekeep, V1-V3: Early Visual Cortex Alignment Shields Vision-Language Models from Sycophantic Manipulation

2026-04-15

AI Summary

This study investigates whether alignment with the human visual system enhances the robustness of vision-language models against flattery-based manipulation in high-stakes scenarios. Evaluating 12 open-source models, the authors measure neural alignment via fMRI prediction accuracy and assess susceptibility to 76,800 gaslighting prompts spanning multiple difficulty levels. They report a significant negative correlation (r = −0.441) between alignment with early visual cortex regions (V1–V3) and manipulability, with the strongest effect observed for denial-type attacks (r = −0.597, p = 0.040). Notably, no such relationship emerges for higher-level semantic brain regions. These findings suggest that fidelity to low-level visual representations serves as a critical anchor against linguistic override, offering a novel pathway toward more robust vision-language systems.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Safety and Robustness
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Vision-language models are increasingly deployed in high-stakes settings, yet their susceptibility to sycophantic manipulation remains poorly understood, particularly in relation to how these models represent visual information internally. Whether models whose visual representations more closely mirror human neural processing are also more resistant to adversarial pressure is an open question with implications for both neuroscience and AI safety. We investigate this question by evaluating 12 open-weight vision-language models spanning 6 architecture families and a 40$\times$ parameter range (256M--10B) along two axes: brain alignment, measured by predicting fMRI responses from the Natural Scenes Dataset across 8 human subjects and 6 visual cortex regions of interest, and sycophancy, measured through 76,800 two-turn gaslighting prompts spanning 5 categories and 10 difficulty levels. Region-of-interest analysis reveals that alignment specifically in early visual cortex (V1--V3) is a reliable negative predictor of sycophancy ($r = -0.441$, BCa 95\% CI $[-0.740, -0.031]$), with all 12 leave-one-out correlations negative and the strongest effect for existence denial attacks ($r = -0.597$, $p = 0.040$). This anatomically specific relationship is absent in higher-order category-selective regions, suggesting that faithful low-level visual encoding provides a measurable anchor against adversarial linguistic override in vision-language models. We release our code on \href{https://github.com/aryashah2k/Gaslight-Gatekeep-Sycophantic-Manipulation}{GitHub} and dataset on \href{https://huggingface.co/datasets/aryashah00/Gaslight-Gatekeep-V1-V3}{Hugging Face}

Problem

• sycophantic manipulation
• vision-language models
• brain alignment
• early visual cortex
• adversarial pressure

Innovation

• brain alignment
• sycophantic manipulation
• early visual cortex
• vision-language models
• adversarial robustness