Neural Encoding and Decoding at Scale

2025-04-11

AI Summary

Existing neural encoding/decoding models support only unidirectional mapping (behavior→neural or neural→behavior), failing to capture the bidirectional causal relationships between neural activity and behavior. Method: We propose the first large-scale, multi-animal, multi-task neural-behavioral bidirectional modeling framework, built upon a multimodal Transformer architecture. It introduces a novel joint masking pretraining strategy across neural, behavioral, and cross-modal modalities, enabling unified encoding and decoding. The model is pretrained at scale on the International Brain Laboratory (IBL) repeated-sites dataset and supports cross-animal transfer fine-tuning—without requiring explicit brain-region annotations. Contribution/Results: Pretrained embeddings spontaneously acquire brain-region prediction capability. In the multi-animal pretraining + novel-animal fine-tuning paradigm, our model achieves state-of-the-art performance on both encoding and decoding tasks, demonstrating strong potential as a general-purpose foundational model for brain-behavior research.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Recent work has demonstrated that large-scale, multi-animal models are powerful tools for characterizing the relationship between neural activity and behavior. Current large-scale approaches, however, focus exclusively on either predicting neural activity from behavior (encoding) or predicting behavior from neural activity (decoding), limiting their ability to capture the bidirectional relationship between neural activity and behavior. To bridge this gap, we introduce a multimodal, multi-task model that enables simultaneous Neural Encoding and Decoding at Scale (NEDS). Central to our approach is a novel multi-task-masking strategy, which alternates between neural, behavioral, within-modality, and cross-modality masking. We pretrain our method on the International Brain Laboratory (IBL) repeated site dataset, which includes recordings from 83 animals performing the same visual decision-making task. In comparison to other large-scale models, we demonstrate that NEDS achieves state-of-the-art performance for both encoding and decoding when pretrained on multi-animal data and then fine-tuned on new animals. Surprisingly, NEDS's learned embeddings exhibit emergent properties: even without explicit training, they are highly predictive of the brain regions in each recording. Altogether, our approach is a step towards a foundation model of the brain that enables seamless translation between neural activity and behavior.

Problem

• Bidirectional neural-behavior relationship modeling
• Simultaneous neural encoding and decoding
• Multi-animal neural data generalization

Innovation

• Multimodal multi-task model for neural-behavioral translation
• Novel multi-task-masking strategy for diverse data
• Pretraining on multi-animal data enhances performance