MindAlign: Decoding Inner Speech from fMRI Signals via Multimodal Embedding Alignment under Limited Data

2026-06-15

AI Summary

This work addresses the challenges of decoding silent, internal speech—namely the absence of overt output, scarce neural data, and substantial inter-subject variability—by introducing MindAlign, a decoupled two-stage brain-to-language framework. In the first stage, fMRI signals are mapped into a shared multimodal semantic space to produce a semantic sketch. The second stage leverages visual context and a prompting mechanism to guide a frozen multimodal large language model for open-ended text generation, eliminating the need for language model fine-tuning. This approach enables cross-subject decoding without subject-specific adaptation and significantly outperforms both fMRI-only and random baselines. The results demonstrate that neural signals encode semantic information beyond image priors and establish new advances in scalability and generalization for brain-to-text decoding.

Abstract

Decoding inner speech from non-invasive brain signals remains a fundamental challenge due to the absence of overt linguistic output, limited training data, and large inter-subject variability. Existing brain-to-text approaches often rely on task-specific decoder fine-tuning, which restricts scalability and complicates adaptation to new participants. We propose MindAlign, a decoupled two-stage brain-to-language framework that enables open-ended text generation from fMRI signals without modifying the underlying language model. The first stage learns a subject-specific neural-semantic alignment that maps fMRI activity into a shared multimodal semantic space, extracting a latent semantic sketch of the internally generated sentence. The second stage integrates this sketch with visual context to prompt a frozen multimodal language model for free-form generation. Experiments on fMRI data collected during silent image description demonstrate that the proposed approach consistently outperforms fMRI-only and random baselines. We further show that the learned semantic-to-language projection can generalize across subjects, enabling effective decoding when paired with subject-specific neural alignment. These results indicate that neural signals modulate semantic content beyond image-driven priors, supporting a scalable and modular direction for brain-to-text decoding.

Problem

• inner speech decoding
• fMRI
• limited data
• inter-subject variability
• brain-to-text

Innovation

• inner speech decoding
• fMRI
• multimodal embedding alignment
• zero-shot brain-to-text
• subject-generalizable decoding