Dense ReLU Neural Networks for Temporal-spatial Model

2024-11-15

arXiv.org

AI Summary

This paper addresses nonparametric modeling of high-dimensional spatiotemporal dependent data, tackling the challenges of unknown intrinsic structure and severe curse of dimensionality. Methodologically, it proposes a modeling paradigm based on dense fully connected ReLU networks, incorporating the manifold hypothesis to capture low-dimensional embedded structures and establishing, for the first time, a non-asymptotic convergence rate for such dense ReLU networks—extending spatiotemporal dependence modeling to short-range dependence scenarios. Theoretically, it rigorously proves that the estimator achieves the minimax optimal convergence rate over generalized Hölder classes. Simulation studies demonstrate that the method consistently outperforms existing approaches in both prediction accuracy and robustness across diverse synthetic response functions. Collectively, this work provides a statistically grounded and practically viable neural-network-based framework for high-dimensional spatiotemporal data analysis.

Technology Category

• Machine Learning: Learning with Manifolds
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks

Abstract

In this paper, we focus on fully connected deep neural networks utilizing the Rectified Linear Unit (ReLU) activation function for nonparametric estimation. We derive non-asymptotic bounds that lead to convergence rates, addressing both temporal and spatial dependence in the observed measurements. By accounting for dependencies across time and space, our models better reflect the complexities of real-world data, enhancing both predictive performance and theoretical robustness. We also tackle the curse of dimensionality by modeling the data on a manifold, exploring the intrinsic dimensionality of high-dimensional data. We broaden existing theoretical findings of temporal-spatial analysis by applying them to neural networks in more general contexts and demonstrate that our proof techniques are effective for models with short-range dependence. Our empirical simulations across various synthetic response functions underscore the superior performance of our method, outperforming established approaches in the existing literature. These findings provide valuable insights into the strong capabilities of dense neural networks (Dense NN) for temporal-spatial modeling across a broad range of function classes.

Problem

• High-dimensional Data
• Temporal and Spatial Correlation
• Complex Data Structure

Innovation

• Fully Connected Dense ReLU Networks
• Temporal and Spatial Data Analysis
• Enhanced Learning Rate with Data Correlation