One Brain, Omni Modalities: Towards Unified Non-Invasive Brain Decoding with Large Language Models

2026-02-24

AI Summary

This work addresses the long-standing challenge of analyzing non-invasive brain signals—such as EEG/MEG and fMRI—in isolation due to their modality heterogeneity, which has impeded a unified understanding of neural activity. The authors propose NOBEL, the first framework within a large language model paradigm to achieve a unified representation of electromagnetic (EEG/MEG) and metabolic (fMRI) signals. By integrating an EEG/MEG joint encoder, a dual-path fMRI processor, and a multimodal semantic embedding alignment mechanism, NOBEL maps neural responses and external sensory stimuli into a shared semantic space, enabling cross-modal fusion and causal decoding. Experiments demonstrate robust performance on unimodal tasks, significant improvements over baselines in multimodal fusion, and successful visual semantic decoding on the NSD and HAD datasets, thereby validating the causal link between sensory stimuli and neural responses.

Technology Category

• Machine Learning: Multimodal Learning
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Humans and AI: Brain-Sensing and Analysis

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Deciphering brain function through non-invasive recordings requires synthesizing complementary high-frequency electromagnetic (EEG/MEG) and low-frequency metabolic (fMRI) signals. However, despite their shared neural origins, extreme discrepancies have traditionally confined these modalities to isolated analysis pipelines, hindering a holistic interpretation of brain activity. To bridge this fragmentation, we introduce \textbf{NOBEL}, a \textbf{n}euro-\textbf{o}mni-modal \textbf{b}rain-\textbf{e}ncoding \textbf{l}arge language model (LLM) that unifies these heterogeneous signals within the LLM's semantic embedding space. Our architecture integrates a unified encoder for EEG and MEG with a novel dual-path strategy for fMRI, aligning non-invasive brain signals and external sensory stimuli into a shared token space, then leverages an LLM as a universal backbone. Extensive evaluations demonstrate that NOBEL serves as a robust generalist across standard single-modal tasks. We also show that the synergistic fusion of electromagnetic and metabolic signals yields higher decoding accuracy than unimodal baselines, validating the complementary nature of multiple neural modalities. Furthermore, NOBEL exhibits strong capabilities in stimulus-aware decoding, effectively interpreting visual semantics from multi-subject fMRI data on the NSD and HAD datasets while uniquely leveraging direct stimulus inputs to verify causal links between sensory signals and neural responses. NOBEL thus takes a step towards unifying non-invasive brain decoding, demonstrating the promising potential of omni-modal brain understanding.

Problem

• non-invasive brain decoding
• multi-modal integration
• EEG/MEG
• fMRI
• brain-computer interface

Innovation

• omni-modal brain decoding
• large language model
• EEG/MEG-fMRI fusion
• neural encoding
• stimulus-aware decoding