Latent Space Energy-based Neural ODEs

2024-09-05

arXiv.org

AI Summary

This work addresses the poor generalization in continuous-time series modeling caused by entanglement between dynamic evolution and static factors. We propose a novel neural ODE–Energy-Based Model (EBM) joint framework: a neural ODE implicitly models the continuous-time latent dynamics, while a learnable EBM prior is explicitly embedded into the latent space—marking the first integration of such priors to disentangle dynamic states from static factors. Coupled with a neural emission model and MCMC-based approximate inference, the framework enables end-to-end maximum-likelihood training. Evaluated on oscillatory systems, video frame sequences, and real-world MuJoCo dynamics data, our method significantly outperforms existing approaches. Notably, it achieves superior long-horizon prediction performance under unseen dynamical parameters, demonstrating strong out-of-distribution generalization. This work establishes a new paradigm for continuous-time representation learning by unifying principled differential equation modeling with expressive, structured latent priors.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Reasoning under Uncertainty: Relational Probabilistic Models
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This paper introduces novel deep dynamical models designed to represent continuous-time sequences. Our approach employs a neural emission model to generate each data point in the time series through a non-linear transformation of a latent state vector. The evolution of these latent states is implicitly defined by a neural ordinary differential equation (ODE), with the initial state drawn from an informative prior distribution parameterized by an Energy-based model (EBM). This framework is extended to disentangle dynamic states from underlying static factors of variation, represented as time-invariant variables in the latent space. We train the model using maximum likelihood estimation with Markov chain Monte Carlo (MCMC) in an end-to-end manner. Experimental results on oscillating systems, videos and real-world state sequences (MuJoCo) demonstrate that our model with the learnable energy-based prior outperforms existing counterparts, and can generalize to new dynamic parameterization, enabling long-horizon predictions.

Problem

• Model continuous-time sequences dynamically
• Disentangle dynamic and static latent factors
• Enable long-horizon predictions with learnable priors

Innovation

• Neural ODEs for dynamics
• Energy-based model priors
• End-to-end MCMC training