Survey on Algorithms for multi-index models

2025-04-07

AI Summary

This paper addresses efficient estimation of the index subspace in multi-index models under Gaussian covariates, focusing on designing and analyzing polynomial-time algorithms. We propose two novel approaches: a nonparametric gradient span estimator and a neural-network-based gradient descent fitting algorithm. We systematically characterize the distributional assumptions and sample complexity required for consistency of both methods. Key contributions include: (i) the first quantitative characterization of the substantial gap between the sample complexity of the fastest existing algorithms and the information-theoretic lower bound; (ii) a unified statistical–computational trade-off analysis comparing the two paradigms; and (iii) a precise delineation of the fundamental boundaries among distributional assumptions, computational efficiency, and statistical accuracy for mainstream estimators. Collectively, these results provide a theoretical framework for designing subspace estimators that are simultaneously computationally efficient and statistically robust.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Graphical Models

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

We review the literature on algorithms for estimating the index space in a multi-index model. The primary focus is on computationally efficient (polynomial-time) algorithms in Gaussian space, the assumptions under which consistency is guaranteed by these methods, and their sample complexity. In many cases, a gap is observed between the sample complexity of the best known computationally efficient methods and the information-theoretical minimum. We also review algorithms based on estimating the span of gradients using nonparametric methods, and algorithms based on fitting neural networks using gradient descent

Problem

• Review algorithms for estimating multi-index model index space
• Analyze computationally efficient methods in Gaussian space
• Compare sample complexity gaps between methods

Innovation

• Polynomial-time algorithms in Gaussian space
• Nonparametric gradient span estimation methods
• Neural network fitting via gradient descent