DeNOTS: Stable Deep Neural ODEs for Time Series

2024-08-15

AI Summary

Neural ODEs suffer from numerical instability during long-horizon time-series modeling due to expanding integration intervals, limiting their expressiveness and practicality. To address this, we propose a stable neural ODE framework with adaptive negative feedback. First, we theoretically establish and guarantee integration stability of the ODE system. Second, we introduce a learnable negative feedback mechanism that endows the model with forgetting capability and robustness to missing values. Third, we eliminate the constraint of time normalization, enabling modeling of continuous-time dynamics over arbitrary durations. Our method integrates stability analysis, time-aware parameterization, and differentiable ODE solvers (e.g., Dopri5). Evaluated on three public time-series benchmarks, it outperforms state-of-the-art baselines—including state-space models and neural controlled differential equations—by up to 20% on downstream tasks, significantly improving long-term forecasting accuracy and irregularly sampled time-series modeling.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Neural ODEs are a prominent branch of methods designed to capture the temporal evolution of complex time-stamped data. Their idea is to solve an ODE with Neural Network-defined dynamics, which take the immediate parameters of the observed system into account. However, larger integration intervals cause instability, which forces most modern methods to normalize time to $[0, 1]$. We provably stabilize these models by introducing an adaptive negative feedback mechanism. This modification allows for longer integration, which in turn implies higher expressiveness, mirroring the behaviour of increasing depth in conventional Neural Networks.Additionally, it provides intriguing theoretical properties: forgetfulness and missing-value robustness. For three open datasets, our method obtains up to 20% improvements in downstream quality if compared to existing baselines, including State Space Models and Neural~CDEs.

Problem

• Stabilize Neural ODEs for longer time integration
• Enhance expressiveness in modeling temporal evolution
• Improve robustness to missing values in time series

Innovation

• Stabilizes Neural ODEs with adaptive feedback
• Enables longer integration for higher expressiveness
• Improves downstream quality by 20 percent