Efficient Training of Neural Stochastic Differential Equations by Matching Finite Dimensional Distributions

2024-10-04

arXiv.org

AI Summary

Neural stochastic differential equations (Neural SDEs) suffer from training instability, mode collapse, and high computational complexity—e.g., signature kernel methods incur $O(D^2)$ cost per iteration. To address these issues, this paper proposes a novel training paradigm based on finite-dimensional distribution matching (FDM). Its core innovation is the first construction of a strictly proper scoring rule for continuous-time Markov processes, leveraging the Markov property of SDEs to design a differentiable, unbiased loss function. Unlike adversarial GAN-style training, our approach eliminates gradient oscillations and instability while reducing per-iteration complexity to $O(D)$. Extensive experiments on benchmark time-series generation tasks demonstrate that our method achieves superior sample quality and training stability compared to state-of-the-art GAN-based and signature kernel-based approaches.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Neural Stochastic Differential Equations (Neural SDEs) have emerged as powerful mesh-free generative models for continuous stochastic processes, with critical applications in fields such as finance, physics, and biology. Previous state-of-the-art methods have relied on adversarial training, such as GANs, or on minimizing distance measures between processes using signature kernels. However, GANs suffer from issues like instability, mode collapse, and the need for specialized training techniques, while signature kernel-based methods require solving linear PDEs and backpropagating gradients through the solver, whose computational complexity scales quadratically with the discretization steps. In this paper, we identify a novel class of strictly proper scoring rules for comparing continuous Markov processes. This theoretical finding naturally leads to a novel approach called Finite Dimensional Matching (FDM) for training Neural SDEs. Our method leverages the Markov property of SDEs to provide a computationally efficient training objective. This scoring rule allows us to bypass the computational overhead associated with signature kernels and reduces the training complexity from $O(D^2)$ to $O(D)$ per epoch, where $D$ represents the number of discretization steps of the process. We demonstrate that FDM achieves superior performance, consistently outperforming existing methods in terms of both computational efficiency and generative quality.

Problem

• Efficient training of Neural SDEs without adversarial methods
• Reducing computational complexity from O(D²) to O(D)
• Improving generative quality and stability in Neural SDEs

Innovation

• Novel scoring rules for Markov processes
• Finite Dimensional Matching training method
• Reduces training complexity to linear scale