Dynamical modeling of nonlinear latent factors in multiscale neural activity with real-time inference

2025-12-13

AI Summary

To address challenges in real-time decoding of multimodal neural signals—such as heterogeneous sampling rates across modalities (e.g., spike trains and field potentials), distributional shifts, and dynamic missingness—this paper proposes the first real-time inference-capable multiscale nonlinear fusion framework. Methodologically, we design intra-modal dynamics-driven multiscale encoders integrated with a recurrent latent-variable dynamics backbone, enabling temporal alignment, missingness-aware feature alignment, and modality-specific probabilistic decoding. Our key contribution lies in unifying variational multiscale encoding with recurrent dynamical modeling within a single real-time decoding architecture. Evaluated on three real-world multiscale brain datasets, our approach significantly improves target decoding accuracy, consistently outperforming both linear models and state-of-the-art nonlinear multimodal baselines across all benchmarks.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Real-time decoding of target variables from multiple simultaneously recorded neural time-series modalities, such as discrete spiking activity and continuous field potentials, is important across various neuroscience applications. However, a major challenge for doing so is that different neural modalities can have different timescales (i.e., sampling rates) and different probabilistic distributions, or can even be missing at some time-steps. Existing nonlinear models of multimodal neural activity do not address different timescales or missing samples across modalities. Further, some of these models do not allow for real-time decoding. Here, we develop a learning framework that can enable real-time recursive decoding while nonlinearly aggregating information across multiple modalities with different timescales and distributions and with missing samples. This framework consists of 1) a multiscale encoder that nonlinearly aggregates information after learning within-modality dynamics to handle different timescales and missing samples in real time, 2) a multiscale dynamical backbone that extracts multimodal temporal dynamics and enables real-time recursive decoding, and 3) modality-specific decoders to account for different probabilistic distributions across modalities. In both simulations and three distinct multiscale brain datasets, we show that our model can aggregate information across modalities with different timescales and distributions and missing samples to improve real-time target decoding. Further, our method outperforms various linear and nonlinear multimodal benchmarks in doing so.

Problem

• Real-time decoding from multiscale neural modalities
• Handling different timescales and missing data
• Nonlinear aggregation across diverse probabilistic distributions

Innovation

• Multiscale encoder handles different timescales and missing data
• Multiscale dynamical backbone extracts multimodal temporal dynamics
• Modality-specific decoders account for different probabilistic distributions