BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language

2026-06-29

AI Summary

Existing brain signal encoding and decoding methods typically treat tasks in isolation, relying on unimodal alignment and external priors while neglecting the brain’s inherent multimodal integration. This work proposes BrainJanus, the first unified framework that jointly models brain activity, vision, and language. It introduces a unified brain tokenizer to discretize continuous neural activity into tokens, aligns all modalities within a shared Omni representation space, and employs an all-in-one autoregressive architecture enabling arbitrary-to-arbitrary generation. Without task-specific designs, BrainJanus achieves zero-shot generalization while preserving interpretable neuroanatomical topology. Experiments demonstrate that BrainJanus outperforms existing approaches across multiple benchmarks, offering both strong cross-modal generative capabilities and significant potential for neuroscience interpretation.

Abstract

Modeling the bidirectional correspondence between external sensory stimuli and internal neural activity has emerged as a critical frontier in neuroscience. However, existing approaches predominantly treat brain encoding and decoding as isolated tasks, relying heavily on unimodal alignment and external priors while overlooking the brain's intrinsic nature as a multimodal integration system. To address these limitations, we propose BrainJanus, the first unified brain model that integrates brain, vision, and language within a single framework. Specifically, we introduce a Unified Brain Tokenizer to quantize continuous neural dynamics into discrete tokens aligned with visual and linguistic representations in a shared Omni space. Building on this, we utilize an All-in-One autoregressive architecture that leverages next-token prediction to enable seamless any-to-any generation, which encompasses image-to-brain and text-to-brain encoding, and brain-to-image and brain-to-text decoding. Extensive experiments demonstrate that BrainJanus achieves superior performance across diverse benchmarks. Furthermore, our framework exhibits zero-shot generalization and preserves interpretable biological topography, highlighting its potential as a general-purpose brain modeling paradigm. The code is available at \href{https://github.com/HaitaoWuTJU/BrainJanus}{GitHub}.

Problem

• brain encoding
• brain decoding
• multimodal integration
• neural activity
• sensory stimuli

Innovation

• unified brain model
• multimodal integration
• brain tokenizer
• any-to-any generation
• zero-shot generalization