Bridging the Vision-Brain Gap with an Uncertainty-Aware Blur Prior

2025-03-06

AI Summary

This paper addresses systematic mismatches (e.g., perceptual biases, cognitive dynamics) and stochastic mismatches (e.g., acquisition noise) between brain signals and visual stimuli in zero-shot brain-to-image retrieval. To this end, we propose Uncertainty-Aware Blurred Priors (UBP)—the first method to explicitly model uncertainty in paired data mismatches. UBP estimates brain–image matching uncertainty and adaptively blurs high-frequency image details to mitigate mismatch effects, while integrating pretrained visual features into a cross-modal alignment and retrieval framework. Evaluated on standard benchmarks, UBP achieves top-1 and top-5 retrieval accuracies of 50.9% and 79.7%, respectively—surpassing the state-of-the-art by 13.7 and 9.8 percentage points. The approach significantly improves model generalization and robustness under distributional shifts and noisy inputs.

Technology Category

• Computer Vision: Image and Video Retrieval
• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Can our brain signals faithfully reflect the original visual stimuli, even including high-frequency details? Although human perceptual and cognitive capacities enable us to process and remember visual information, these abilities are constrained by several factors, such as limited attentional resources and the finite capacity of visual memory. When visual stimuli are processed by human visual system into brain signals, some information is inevitably lost, leading to a discrepancy known as the extbf{System GAP}. Additionally, perceptual and cognitive dynamics, along with technical noise in signal acquisition, degrade the fidelity of brain signals relative to the visual stimuli, known as the extbf{Random GAP}. When encoded brain representations are directly aligned with the corresponding pretrained image features, the System GAP and Random GAP between paired data challenge the model, requiring it to bridge these gaps. However, in the context of limited paired data, these gaps are difficult for the model to learn, leading to overfitting and poor generalization to new data. To address these GAPs, we propose a simple yet effective approach called the extbf{Uncertainty-aware Blur Prior (UBP)}. It estimates the uncertainty within the paired data, reflecting the mismatch between brain signals and visual stimuli. Based on this uncertainty, UBP dynamically blurs the high-frequency details of the original images, reducing the impact of the mismatch and improving alignment. Our method achieves a top-1 accuracy of extbf{50.9%} and a top-5 accuracy of extbf{79.7%} on the zero-shot brain-to-image retrieval task, surpassing previous state-of-the-art methods by margins of extbf{13.7%} and extbf{9.8%}, respectively. Code is available at href{https://github.com/HaitaoWuTJU/Uncertainty-aware-Blur-Prior}{GitHub}.

Problem

• Bridging the gap between brain signals and visual stimuli.
• Addressing information loss due to perceptual and cognitive constraints.
• Improving alignment accuracy in brain-to-image retrieval tasks.

Innovation

• Uncertainty-aware Blur Prior (UBP) introduced
• Dynamic blurring of high-frequency image details
• Improved brain-to-image retrieval accuracy significantly