Has the Deep Neural Network learned the Stochastic Process? An Evaluation Viewpoint

2024-02-23

AI Summary

Deep neural networks (DNNs) often achieve high predictive accuracy on complex stochastic systems without genuinely capturing their underlying stochastic dynamics—a fundamental “prediction accuracy ≠ true understanding” gap. Method: We propose F2SP (Fidelity to Stochastic Process), the first formal evaluation criterion to assess whether a model faithfully encodes the latent stochastic evolution, rather than merely fitting observed trajectories. We rigorously define and axiomatize F2SP; prove that Expected Calibration Error (ECE) is the unique existing metric satisfying its necessary conditions; and construct a theoretically grounded, verifiable evaluation framework rooted in stochastic process theory. Results: We systematically validate our framework on synthetic models (wildfire spread, host–pathogen dynamics, stock markets) and real-world wildfire data. Results demonstrate that ECE uniquely and effectively quantifies F2SP fidelity, whereas conventional error- or threshold-based metrics exhibit intrinsic limitations. This work establishes a novel evaluation paradigm and rigorous theoretical foundation for DNN modeling of complex stochastic systems.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Calibration & Uncertainty Quantification
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution

Abstract

This paper presents the first systematic study of evaluating Deep Neural Networks (DNNs) designed to forecast the evolution of stochastic complex systems. We show that traditional evaluation methods like threshold-based classification metrics and error-based scoring rules assess a DNN's ability to replicate the observed ground truth but fail to measure the DNN's learning of the underlying stochastic process. To address this gap, we propose a new evaluation criterion called Fidelity to Stochastic Process (F2SP), representing the DNN's ability to predict the system property Statistic-GT--the ground truth of the stochastic process--and introduce an evaluation metric that exclusively assesses F2SP. We formalize F2SP within a stochastic framework and establish criteria for validly measuring it. We formally show that Expected Calibration Error (ECE) satisfies the necessary condition for testing F2SP, unlike traditional evaluation methods. Empirical experiments on synthetic datasets, including wildfire, host-pathogen, and stock market models, demonstrate that ECE uniquely captures F2SP. We further extend our study to real-world wildfire data, highlighting the limitations of conventional evaluation and discuss the practical utility of incorporating F2SP into model assessment. This work offers a new perspective on evaluating DNNs modeling complex systems by emphasizing the importance of capturing the underlying stochastic process.

Problem

• Deep Neural Networks
• Complex System Predictions
• Stochastic Pattern Understanding

Innovation

• Fidelity of Stochastic Processes (F2SP)
• Expected Calibration Error (ECE)
• Deep Neural Network Predictions