Task-guided cross-subject latent alignment: a multi-encoder-decoder VAE

2026-06-14

AI Summary

This work addresses the challenge of cross-subject neural alignment in naturalistic settings where shared stimulus exposure is unavailable—a limitation of conventional methods. To overcome this, the authors propose a Multi-Encoder–Decoder Variational Autoencoder (MED-VAE) that leverages a pretrained artificial neural network (ANN) as a universal semantic scaffold to construct a shared latent space with meaningful semantic structure, without requiring overlapping stimuli across subjects. This approach achieves, for the first time, effective cross-subject neural alignment and reconstruction in the absence of shared stimuli. Evaluated on the Natural Scenes Dataset, MED-VAE significantly outperforms existing methods in alignment accuracy and generalization, and successfully enables downstream applications such as cross-subject image decoding.

Abstract

Aligning neural activity across subjects offers the promise of discovering shared computational principles and generalizable decoders. However, traditional alignment methods require shared stimuli across subjects, a constraint that limits applicability to naturalistic paradigms with limited or non-overlapping data. We introduce a Multi-Encoder-Decoder Variational Autoencoder (MED-VAE) that achieves cross-subject alignment without shared stimuli by anchoring representations to a common scaffold provided by a pretrained ANN. Using the Natural Scenes Dataset, we show that MED-VAE creates common latent spaces with superior semantic organisation, achieving higher cross-subject alignment than common methods while maintaining robust generalisation to held-out stimuli where traditional methods degrade. Reconstructing from these common spaces back to each subject's original neural space, MED-VAE preserves equal stimulus-driven signal in its cross-subject latent space. Finally, we show that this superior alignment directly enables cross-subject neural prediction, as demonstrated via cross-subject image decoding. In summary, we introduce a framework to identify generalisable common subspaces for cross-subject predictions and downstream tasks, demonstrated here for visual cortex responses to static images.

Problem

• cross-subject alignment
• neural decoding
• shared latent space
• naturalistic stimuli
• generalizable representation

Innovation

• cross-subject alignment
• variational autoencoder
• pretrained ANN scaffold
• neural decoding
• latent space