From "What-If" to "What-Is": Counterfactual Thinking-Inspired Semantic Alignment for Visual Brain Decoding

2026-08-15

AI Summary

This study addresses the challenges of evaluating and aligning semantic deviations in visual brain decoding for image reconstruction by proposing ConceptAlign, a counterfactual semantic alignment framework. Leveraging large language models to generate near-miss samples offline, this method employs a margin loss to refine decoding representations within a frozen text embedding space, enabling fine-grained semantic boundary learning without inference-time overhead. Furthermore, it establishes a three-tiered semantic evaluation system. Experiments on the NSD dataset demonstrate that ConceptAlign significantly enhances the semantic accuracy, counterfactual discriminability, and representational alignment of reconstructed images, effectively resolving semantic ambiguity in brain decoding tasks.

Abstract

Visual brain decoding reconstructs visual content perceived by a person from neural measurements such as fMRI, providing a computational approach to studying how visual information is represented in the brain. Recent multimodal representations and diffusion priors have improved reconstruction realism. However, visually plausible reconstructions may contain incorrect objects, attributes, or relations because a strong generative prior can complete content not sufficiently specified by the decoded representation. Conventional reconstruction metrics mainly assess the final image and may therefore obscure such semantic errors. We propose ConceptAlign, a counterfactual semantic alignment framework for visual brain decoding. ConceptAlign pools decoded visual tokens and projects them into a frozen text-embedding space, aligning the representation with the ground-truth caption while separating it from scene-preserving near-miss alternatives. Generated offline by an LLM, these alternatives modify one critical object, attribute, or relation while retaining the scene. A margin-based objective learns fine-grained semantic boundaries between the observed stimulus and plausible but incorrect interpretations without requiring LLM calls during inference. We introduce a systematic three-level semantic evaluation framework covering foundational discriminability, counterfactual description discrimination, and representational geometry. Experiments on the Natural Scenes Dataset show that ConceptAlign improves reconstruction measures, counterfactual semantic discrimination, and representational alignment over the MindEye2 backbone. Matched negative-source ablations, independent LLM and human-written alternatives, and human evaluation support the effectiveness and robustness of the supervision, with favorable patterns in fine-grained conflicts, limited-data decoding, and cross-subject structure.

Problem

• Visual Brain Decoding
• Semantic Alignment
• Reconstruction Metrics
• Generative Prior

Innovation

• Counterfactual Semantic Alignment
• Visual Brain Decoding
• ConceptAlign
• Margin-based Objective
• Semantic Evaluation Framework