Conditional regression for the Nonlinear Single-Variable Model

2024-11-14

arXiv.org

AI Summary

This paper studies high-dimensional nonlinear single-index regression models of the form $F(X) = f(Pi_gamma X)$, where $Pi_gamma$ denotes the nearest-point projection of the input $X$ onto an unknown regular curve $gamma$, and $f$ is an unknown scalar function—constituting a nonlinear generalization of the single-index model. To overcome the curse of dimensionality that plagues conventional methods, we develop the first nonparametric estimation theory for this geometric projection structure that avoids dimensional dependence. Under mild assumptions—including coarse monotonicity of $f$—our estimator achieves the one-dimensional optimal minimax convergence rate $O(n^{-1/3})$. Its computational complexity is merely $O(d^2 n log n)$, with all constants polynomially bounded in dimension $d$. The key innovation lies in integrating conditional regression, geometric projection analysis, and adaptive neighborhood construction, thereby circumventing reliance on linear structure while unifying statistical optimality and computational feasibility.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Learning with Manifolds
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Several statistical models for regression of a function $F$ on $mathbb{R}^d$ without the statistical and computational curse of dimensionality exist, for example by imposing and exploiting geometric assumptions on the distribution of the data (e.g. that its support is low-dimensional), or strong smoothness assumptions on $F$, or a special structure $F$. Among the latter, compositional models assume $F=fcirc g$ with $g$ mapping to $mathbb{R}^r$ with $rll d$, have been studied, and include classical single- and multi-index models and recent works on neural networks. While the case where $g$ is linear is rather well-understood, much less is known when $g$ is nonlinear, and in particular for which $g$'s the curse of dimensionality in estimating $F$, or both $f$ and $g$, may be circumvented. In this paper, we consider a model $F(X):=f(Pi_gamma X) $ where $Pi_gamma:mathbb{R}^d o[0, m{len}_gamma]$ is the closest-point projection onto the parameter of a regular curve $gamma: [0, m{len}_gamma] omathbb{R}^d$ and $f:[0, m{len}_gamma] omathbb{R}^1$. The input data $X$ is not low-dimensional, far from $gamma$, conditioned on $Pi_gamma(X)$ being well-defined. The distribution of the data, $gamma$ and $f$ are unknown. This model is a natural nonlinear generalization of the single-index model, which corresponds to $gamma$ being a line. We propose a nonparametric estimator, based on conditional regression, and show that under suitable assumptions, the strongest of which being that $f$ is coarsely monotone, it can achieve the $one$-$dimensional$ optimal min-max rate for non-parametric regression, up to the level of noise in the observations, and be constructed in time $mathcal{O}(d^2nlog n)$. All the constants in the learning bounds, in the minimal number of samples required for our bounds to hold, and in the computational complexity are at most low-order polynomials in $d$.

Problem

• Estimating nonlinear single-variable models without dimensionality curse
• Generalizing single-index models to nonlinear curve projections
• Achieving one-dimensional optimal nonparametric regression rates

Innovation

• Nonlinear Single-Variable Model with curve projection
• Conditional regression for dimensionality reduction
• Coarsely monotone function achieves optimal rate