Joint Learning in the Gaussian Single Index Model

2025-05-27

AI Summary

This paper studies joint learning of a one-dimensional projection direction $w^*$ and a univariate function $varphi^*$ in high-dimensional Gaussian settings, i.e., estimating $f(x) = varphi^*(langle w^*, x angle)$—a problem exhibiting both nonconvexity and the interplay between representation learning and nonlinear regression. We propose a gradient-flow-based alternating algorithm and establish, for the first time, its global convergence guarantee, revealing the counterintuitive phenomenon that negatively correlated initializations still converge. To characterize convergence behavior, we introduce the “information index,” which quantifies how Gaussian regularity governs convergence rates. Theoretically, we prove that the convergence rate is determined by the Gaussian regularity of the target function within an appropriately adapted reproducing kernel Hilbert space (RKHS). Empirically, our RKHS-based implementation demonstrates high efficiency, flexibility, and strong robustness, achieving superior generalization performance in high-dimensional sparse structure learning.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Non-convex Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

We consider the problem of jointly learning a one-dimensional projection and a univariate function in high-dimensional Gaussian models. Specifically, we study predictors of the form $f(x)=varphi^star(langle w^star, x angle)$, where both the direction $w^star in mathcal{S}_{d-1}$, the sphere of $mathbb{R}^d$, and the function $varphi^star: mathbb{R} o mathbb{R}$ are learned from Gaussian data. This setting captures a fundamental non-convex problem at the intersection of representation learning and nonlinear regression. We analyze the gradient flow dynamics of a natural alternating scheme and prove convergence, with a rate controlled by the information exponent reflecting the extit{Gaussian regularity} of the function $varphi^star$. Strikingly, our analysis shows that convergence still occurs even when the initial direction is negatively correlated with the target. On the practical side, we demonstrate that such joint learning can be effectively implemented using a Reproducing Kernel Hilbert Space (RKHS) adapted to the structure of the problem, enabling efficient and flexible estimation of the univariate function. Our results offer both theoretical insight and practical methodology for learning low-dimensional structure in high-dimensional settings.

Problem

• Jointly learning projection and function in high-dimensional Gaussian models
• Analyzing gradient flow dynamics for non-convex representation learning
• Implementing efficient estimation using RKHS adapted to problem structure

Innovation

• Joint learning of projection and univariate function
• Gradient flow dynamics with convergence analysis
• RKHS adapted for efficient function estimation