Neuroscience-Inspired Analyses of Visual Interestingness in Multimodal Transformers

2026-05-05

AI Summary

This study investigates whether multimodal Transformers implicitly encode human visual interest mechanisms beyond mere statistical correlations in data. Leveraging the Common Interestingness (CI) scores from Flickr, the authors employ a suite of neuroscience-inspired methods—including linear decoding, representational similarity analysis, generalized discriminability, concept vectors, and sparse autoencoders—to reveal, for the first time under unsupervised conditions, a structured and progressive encoding of CI information within the Qwen3-VL-8B model. CI signals emerge in intermediate layers of the vision Transformer and are progressively amplified through the language module, ultimately becoming linearly decodable from the final embeddings. This finding establishes a direct link between internal representations in large multimodal models and human cognitive mechanisms underlying visual interest.

Abstract

Human attention is the gateway to conscious perception, memory and decision-making. However, its role in modern transformer models remains largely unexplored. As these systems increasingly influence what people see, prefer and buy, the question arises as to whether they encode principles of human interest or merely exploit large-scale correlations. Addressing this issue is crucial for understanding cognition and ensuring the responsible use of AI in communication and marketing. In order to address this issue, the concept of visual interest was examined within the multimodal vision-language-model Qwen3-VL-8B, using a pre-defined Common Interestingness (CI) score derived from large-scale human engagement data on the photo-sharing platform Flickr. Here, we analyzed internal representations across vision and language components using methods from the neurosciences. Our analyses revealed that CI information is linearly decodable from final-layer embeddings, indicating that it is aligned with human-derived measures of visual interestingness. Dimensionality reduction and Generalized Discrimination Value (GDV) analyses demonstrate that CI-related hidden representations emerge in intermediate vision transformer layers and becomes progressively more distinguishable across language model layers. Concept vectors derived using geometric, probe, and Sparse Auto-Encoder based methods converge in higher layers, as confirmed by representational similarity analysis. This indicates a robust and structured encoding of visual interestingness without explicit supervision. Future work will seek to identify shared computational principles linking human brain dynamics and transformer architectures, with the ultimate goal of uncovering the organizing mechanisms that give rise to attention and interest in both biological and artificial systems.

Problem

• visual interestingness
• human attention
• multimodal transformers
• neuroscience-inspired analysis
• Common Interestingness

Innovation

• visual interestingness
• multimodal transformers
• neuroscience-inspired analysis
• Common Interestingness (CI)
• unsupervised representation