Smoothed Analysis for Learning Concepts with Low Intrinsic Dimension

2024-07-01

Annual Conference Computational Learning Theory

AI Summary

Traditional concept learning often encounters computational or statistical hardness barriers under arbitrary input distributions. This work introduces a Gaussian smoothing analysis framework, requiring the learner to approximate only the optimal classifier robust to small Gaussian perturbations—rather than the globally optimal one—thereby circumventing classical hardness obstacles. To our knowledge, this is the first systematic application of smoothed analysis to concept learning. Methodologically, we integrate Gaussian surface area bounds, low-dimensional subspace projections, and margin-based agnostic learning theory. Our approach achieves efficient learnability for low-intrinsic-dimension concept classes—including multi-index models, convex sets, and intersections of $k$ halfspaces—under arbitrary distributions. Key contributions include: (i) the first provably falsifiable learning algorithm for intersections of $k$ halfspaces, with runtime improved from exponential to $k^{mathrm{poly}(log k/(varepsilongamma))}$; and (ii) significant improvements in both sample and computational complexity for margin-based learning.

Technology Category

• Machine Learning: Learning Theory
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In traditional models of supervised learning, the goal of a learner -- given examples from an arbitrary joint distribution on $mathbb{R}^d imes {pm 1}$ -- is to output a hypothesis that is competitive (to within $epsilon$) of the best fitting concept from some class. In order to escape strong hardness results for learning even simple concept classes, we introduce a smoothed-analysis framework that requires a learner to compete only with the best classifier that is robust to small random Gaussian perturbation. This subtle change allows us to give a wide array of learning results for any concept that (1) depends on a low-dimensional subspace (aka multi-index model) and (2) has a bounded Gaussian surface area. This class includes functions of halfspaces and (low-dimensional) convex sets, cases that are only known to be learnable in non-smoothed settings with respect to highly structured distributions such as Gaussians. Surprisingly, our analysis also yields new results for traditional non-smoothed frameworks such as learning with margin. In particular, we obtain the first algorithm for agnostically learning intersections of $k$-halfspaces in time $k^{poly(frac{log k}{epsilon gamma}) }$ where $gamma$ is the margin parameter. Before our work, the best-known runtime was exponential in $k$ (Arriaga and Vempala, 1999).

Problem

• Learning low-dimensional concept classes via smoothed analysis
• Agnostically learning intersections of halfspaces efficiently
• Escaping hardness results with Gaussian perturbation robustness

Innovation

• Smoothed-analysis framework for robust learning
• Learning low-dimensional subspace concepts
• Agnostically learning intersections efficiently