Deep Models, Shallow Alignment: Uncovering the Granularity Mismatch in Neural Decoding

2026-01-29

AI Summary

This work addresses a critical representational granularity mismatch in neural visual decoding: deep vision models, due to their emphasis on semantic invariance, often neglect local texture cues and thus fail to align with the multi-level information present in human brain signals. To resolve this, the authors propose a “shallow alignment” strategy that leverages contrastive learning to align neural responses with intermediate-layer features of pretrained vision models—such as ViT or ResNet—rather than relying solely on their final outputs. This approach is the first to systematically identify and mitigate the granularity mismatch, effectively unlocking the scaling laws of vision models and enabling predictable improvements in decoding performance with increasing model size. Across multiple benchmarks, the method achieves performance gains of 22%–58% over conventional final-layer alignment techniques.

Technology Category

• Computer Vision: Large Vision Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Deep Neural Architectures and Foundation Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Neural visual decoding is a central problem in brain computer interface research, aiming to reconstruct human visual perception and to elucidate the structure of neural representations. However, existing approaches overlook a fundamental granularity mismatch between human and machine vision, where deep vision models emphasize semantic invariance by suppressing local texture information, whereas neural signals preserve an intricate mixture of low-level visual attributes and high-level semantic content. To address this mismatch, we propose Shallow Alignment, a novel contrastive learning strategy that aligns neural signals with intermediate representations of visual encoders rather than their final outputs, thereby striking a better balance between low-level texture details and high-level semantic features. Extensive experiments across multiple benchmarks demonstrate that Shallow Alignment significantly outperforms standard final-layer alignment, with performance gains ranging from 22% to 58% across diverse vision backbones. Notably, our approach effectively unlocks the scaling law in neural visual decoding, enabling decoding performance to scale predictably with the capacity of pre-trained vision backbones. We further conduct systematic empirical analyses to shed light on the mechanisms underlying the observed performance gains.

Problem

• neural decoding
• granularity mismatch
• visual perception
• deep vision models
• neural representations

Innovation

• Shallow Alignment
• neural decoding
• granularity mismatch
• contrastive learning
• visual representation