Scaling laws for decoding images from brain activity

2025-01-25

AI Summary

This study systematically investigates the performance limits and data-dependency laws of single-trial image decoding from non-invasive brain signals (EEG, MEG, 3T/7T fMRI). Leveraging eight public datasets, 84 participants, 498 hours of neural recordings, and 2.3 million brain responses, we establish the largest-scale image decoding benchmark to date. Methodologically, we analyze scaling behavior across modalities, standardize cross-modal signals, model brain–image alignment, and rigorously compare deep learning architectures against linear baselines. Results reveal that decoding accuracy scales logarithmically with per-participant data volume—not participant count—and that deep learning yields the greatest gains in high-noise modalities (e.g., EEG), with no saturation observed. We identify 7T fMRI as the empirical accuracy ceiling, while EEG coupled with deep networks achieves breakthrough single-trial decoding. Core contributions include: (1) uncovering the data-scaling law for non-invasive image decoding; and (2) establishing a modality–algorithm co-optimization paradigm.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Ethical and legal aspects of web-scale data analysis, uses and collection practices
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Generative AI has recently propelled the decoding of images from brain activity. How do these approaches scale with the amount and type of neural recordings? Here, we systematically compare image decoding from four types of non-invasive devices: electroencephalography (EEG), magnetoencephalography (MEG), high-field functional Magnetic Resonance Imaging (3T fMRI) and ultra-high field (7T) fMRI. For this, we evaluate decoding models on the largest benchmark to date, encompassing 8 public datasets, 84 volunteers, 498 hours of brain recording and 2.3 million brain responses to natural images. Unlike previous work, we focus on single-trial decoding performance to simulate real-time settings. This systematic comparison reveals three main findings. First, the most precise neuroimaging devices tend to yield the best decoding performances, when the size of the training sets are similar. However, the gain enabled by deep learning - in comparison to linear models - is obtained with the noisiest devices. Second, we do not observe any plateau of decoding performance as the amount of training data increases. Rather, decoding performance scales log-linearly with the amount of brain recording. Third, this scaling law primarily depends on the amount of data per subject. However, little decoding gain is observed by increasing the number of subjects. Overall, these findings delineate the path most suitable to scale the decoding of images from non-invasive brain recordings.

Problem

• Non-invasive Brain Activity
• Image Decoding
• Deep Learning

Innovation

• Neuroimaging Technologies
• Deep Learning
• Decoding Accuracy