Optimizing fMRI Data Acquisition for Decoding Natural Speech with Limited Participants

2025-05-27

AI Summary

This study addresses the practical bottleneck of limited subject availability (N=8) in fMRI-based speech decoding. Using the Lebel et al. (2023) dataset, we systematically compare single-subject deep scanning against multi-subject joint modeling, employing deep neural networks to decode brain activity during natural speech perception by predicting text embeddings generated by large language models (LLMs). Key findings are: (1) Multi-subject joint training fails to improve decoding accuracy—challenging conventional data-aggregation paradigms; (2) Single-subject deep scanning significantly outperforms cross-subject mixed training; and (3) Decoders exhibit markedly stronger sensitivity to syntactic than semantic features, with performance degrading substantially on stimuli involving complex syntax or high semantic density. These results underscore the necessity and superiority of individualized, deep phenotypic modeling in neuroimaging, offering methodological insights for low-sample-size neural decoding.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We investigate optimal strategies for decoding perceived natural speech from fMRI data acquired from a limited number of participants. Leveraging Lebel et al. (2023)'s dataset of 8 participants, we first demonstrate the effectiveness of training deep neural networks to predict LLM-derived text representations from fMRI activity. Then, in this data regime, we observe that multi-subject training does not improve decoding accuracy compared to single-subject approach. Furthermore, training on similar or different stimuli across subjects has a negligible effect on decoding accuracy. Finally, we find that our decoders better model syntactic than semantic features, and that stories containing sentences with complex syntax or rich semantic content are more challenging to decode. While our results demonstrate the benefits of having extensive data per participant (deep phenotyping), they suggest that leveraging multi-subject for natural speech decoding likely requires deeper phenotyping or a substantially larger cohort.

Problem

• Optimizing fMRI data acquisition for natural speech decoding with limited participants
• Evaluating multi-subject training effectiveness in fMRI-based speech decoding
• Assessing decoder performance on syntactic versus semantic features in fMRI data

Innovation

• Deep neural networks predict text from fMRI
• Single-subject training outperforms multi-subject
• Decoders model syntax better than semantics