Information-theoretic limits and approximate message-passing for high-dimensional time series

2025-01-23

AI Summary

This paper addresses the modeling challenge of high-dimensional time series characterized by non-sparse structure, where the number of features scales proportionally with the sample size and the data contain substantial redundancy. Unlike conventional approaches relying on sparsity assumptions, we operate in a genuinely non-sparse high-dimensional regime. Method: We integrate random linear regression modeling, information-theoretic analysis, and the Vector Approximate Message Passing (VAMP) algorithm. Contribution/Results: We establish, for the first time, the information-theoretic limit in this non-sparse setting—deriving a tight single-letter characterization of the normalized mutual information between observations and latent signals. Theoretically, we prove that VAMP, despite lacking prior guarantees, achieves statistical optimality in this regime. Empirically, VAMP attains the minimum mean-square error (MMSE) limit across diverse configurations, demonstrating robust performance and near-optimal convergence to the theoretical bound—yielding the first solution for high-dimensional time-series recovery grounded in rigorous information theory and consistent with practical implementation.

Technology Category

• Machine Learning: Information Theory
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

High-dimensional time series appear in many scientific setups, demanding a nuanced approach to model and analyze the underlying dependence structure. However, theoretical advancements so far often rely on stringent assumptions regarding the sparsity of the underlying signals. In this contribution, we expand the scope by investigating a high-dimensional time series model wherein the number of features grows proportionally to the number of sampling points, without assuming sparsity in the signal. Specifically, we consider the stochastic regression model and derive a single-letter formula for the normalized mutual information between observations and the signal. We also empirically study the vector approximate message passing (VAMP) algorithm and show that, despite a lack of theoretical guarantees, its performance for inference in our time series model is robust and often statistically optimal.

Problem

• High-dimensional Time Series Analysis
• Complex Information Extraction
• Noise Reduction

Innovation

• High-dimensional Time Series
• VAMP Algorithm
• Optimized Information Boundary