Robustly Learning Monotone Generalized Linear Models via Data Augmentation

2025-02-12

AI Summary

This work studies robust learning of generalized linear models (GLMs) under the agnostic setting with Gaussian covariates, focusing on arbitrary monotone Lipschitz activation functions—a class previously inaccessible to efficient algorithms, which were restricted to narrow subclasses. We propose the first polynomial-time robust learning algorithm: (i) we design a robustified GLMtron framework; (ii) we introduce a data augmentation strategy based on decaying Gaussian noise injection, coupled with (2+ζ)-moment control and structured gradient analysis; and (iii) we achieve constant-factor approximation to the true parameter. Our method is the first to provably extend robust learning to *all* monotone Lipschitz activations with bounded (2+ζ)-th moment, significantly broadening the scope of applicability and resolving a long-standing open problem in this line of research.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We study the task of learning Generalized Linear models (GLMs) in the agnostic model under the Gaussian distribution. We give the first polynomial-time algorithm that achieves a constant-factor approximation for extit{any} monotone Lipschitz activation. Prior constant-factor GLM learners succeed for a substantially smaller class of activations. Our work resolves a well-known open problem, by developing a robust counterpart to the classical GLMtron algorithm (Kakade et al., 2011). Our robust learner applies more generally, encompassing all monotone activations with bounded $(2+zeta)$-moments, for any fixed $zeta>0$ -- a condition that is essentially necessary. To obtain our results, we leverage a novel data augmentation technique with decreasing Gaussian noise injection and prove a number of structural results that may be useful in other settings.

Problem

• Learning Generalized Linear Models robustly
• Achieving constant-factor approximation for monotone activations
• Developing novel data augmentation with Gaussian noise

Innovation

• Polynomial-time algorithm for GLMs
• Data augmentation with Gaussian noise
• Robust counterpart to GLMtron