Learning dynamical systems with hit-and-run random feature maps

2025-01-11

AI Summary

To address low prediction accuracy, complex hyperparameter tuning, and the curse of dimensionality in forecasting high-dimensional chaotic dynamical systems, this paper proposes a data-driven hit-and-run random feature mapping method. Our approach introduces three key innovations: (1) a novel hit-and-run weight sampling strategy that enhances exploration efficiency in feature space; (2) local conditional independence modeling to mitigate statistical sparsity in high dimensions; and (3) integration of tanh-based random features, data-driven initialization, skip connections, and deep feature composition to improve generalization. The method requires tuning only a single hyperparameter yet achieves high-accuracy short-term trajectory prediction and reliable long-term statistical estimation on a 512-dimensional chaotic system. Empirical results demonstrate substantial performance gains over baseline methods—including reservoir computing—which demand extensive hyperparameter optimization.

Technology Category

• Machine Learning: Feature Construction/Reformulation
• Search and Optimization: Metareasoning and Metaheuristics
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We show how random feature maps can be used to forecast dynamical systems with excellent forecasting skill. We consider the tanh activation function and judiciously choose the internal weights in a data-driven manner such that the resulting features explore the nonlinear, non-saturated regions of the activation function. We introduce skip connections and construct a deep variant of random feature maps by combining several units. To mitigate the curse of dimensionality, we introduce localization where we learn local maps, employing conditional independence. Our modified random feature maps provide excellent forecasting skill for both single trajectory forecasts as well as long-time estimates of statistical properties, for a range of chaotic dynamical systems with dimensions up to 512. In contrast to other methods such as reservoir computers which require extensive hyperparameter tuning, we effectively need to tune only a single hyperparameter, and are able to achieve state-of-the-art forecast skill with much smaller networks.

Problem

• Complex Dynamic Systems
• Prediction Accuracy
• Parameter Tuning Complexity

Innovation

• Random Collision Feature Mapping
• Tanh Tool Parameter Adjustment
• Skip Connection Mechanism