Learning Noisy Halfspaces with a Margin: Massart is No Harder than Random

2025-01-16

AI Summary

This work investigates the PAC learnability of γ-margin halfspaces and robust learning of generalized linear models (GLMs) under Massart noise. Addressing the long-standing open question—“Is Massart noise harder to learn than random classification noise?”—we establish, for the first time, that its learnability is no harder than that under random noise. We propose Perspectron, a geometrically aware, margin-sensitive online update framework that unifies learning for both halfspaces and GLMs, achieving an optimal sample complexity of Õ((εγ)⁻²). The algorithm attains classification error η + ε, matching the information-theoretic lower bound. This is the first method under Massart noise to match the sample efficiency of the best-known algorithms for random noise—significantly improving upon prior results—and introduces a new paradigm for robust high-dimensional learning.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

We study the problem of PAC learning $gamma$-margin halfspaces with Massart noise. We propose a simple proper learning algorithm, the Perspectron, that has sample complexity $widetilde{O}((epsilongamma)^{-2})$ and achieves classification error at most $eta+epsilon$ where $eta$ is the Massart noise rate. Prior works [DGT19,CKMY20] came with worse sample complexity guarantees (in both $epsilon$ and $gamma$) or could only handle random classification noise [DDK+23,KIT+23] -- a much milder noise assumption. We also show that our results extend to the more challenging setting of learning generalized linear models with a known link function under Massart noise, achieving a similar sample complexity to the halfspace case. This significantly improves upon the prior state-of-the-art in this setting due to [CKMY20], who introduced this model.

Problem

• Massart Noise
• Gamma-Margin Halfspaces
• Generalized Linear Models

Innovation

• Perspectron
• Massart Noise
• Generalized Linear Models