Beyond Parallel Tracking: Interactive Multi-Feature Fusion Drives Semantic Reconstruction from Non-invasive Brain Recordings

2026-07-13

AI Summary

This study addresses the modality mismatch between non-invasive neural signals and semantic representations, which limits the performance of continuous language reconstruction from brain activity. The authors propose a multi-feature fusion framework that systematically compares linear concatenation and nonlinear cross-attention strategies for the first time, introducing an interactive gating mechanism to jointly integrate static word embeddings (Word2Vec) with dynamic contextual representations (GPT). Experimental results demonstrate that nonlinear fusion based on multi-head cross-attention significantly outperforms alternative approaches, following the hierarchy Cross-Att > Concat > GPT > Word2Vec. These findings highlight the critical role of token-level attributes and context-aware modulation in neural decoding, transcending the limitations of single-feature representations and achieving state-of-the-art performance in non-invasive brain-to-text reconstruction.

Abstract

Continuous semantic reconstruction from non-invasive neural recordings remains limited by the representational mismatch between semantic feature spaces and neural coding patterns, which severely impedes cross-modal alignment between high-noise neural signals and target semantic features. Prior semantic decoders have predominantly relied on static lexical representations or dynamic contextualized representations in isolation. This single-dimension approach inevitably leads to severe information loss, as it fails to account for the human brain's capacity to integrate stable word attributes and dynamic contexts simultaneously.To bridge this gap, this study introduces a multi-feature fusion framework for non-invasive semantic reconstruction, systematically benchmarking two integration approaches: linear Naive Concatenation and non-linear Multi-Head Cross-Attention. Within this framework, our approach complements static lexical representations (W2V) with dynamic contextual representations (GPT) via an interactive gating mechanism to facilitate cooperative processing during language comprehension.Evaluated through extensive semantic reconstruction and text generation experiments, our framework reveals a robust performance hierarchy: Cross-Att > Concat > GPT > W2V. Crucially, the non-linear cross-attention fusion method achieves state-of-the-art performance, demonstrating that neural language decoding benefits from simulating the collaborative modulation between contextual information and core lexical attributes rather than depending on isolated individual features, while also offering a viable non-invasive brain-to-text decoding method.

Problem

• semantic reconstruction
• non-invasive brain recordings
• representational mismatch
• cross-modal alignment
• multi-feature fusion

Innovation

• multi-feature fusion
• cross-modal alignment
• semantic reconstruction
• non-invasive brain decoding
• cross-attention