Batch List-Decodable Linear Regression via Higher Moments

2025-03-12

AI Summary

This work studies robust linear regression under batch-wise corruptions: only an α ∈ (0,1/2) fraction of batches obey the true linear model (i.i.d.), while the rest are arbitrarily corrupted. The goal is to output a small list of candidate parameter vectors, guaranteeing that at least one achieves ℓ₂ estimation error O(α⁻ᵟ⁄²/√n). Methodologically, the paper innovatively integrates Sum-of-Squares (SoS) proofs of high-order moment constraints with iterative reweighting and list pruning, and further SoS-ifies the Marcinkiewicz–Zygmund inequality. With m = poly((dn)¹⁄ᵟ, 1/α) batches, it achieves—for the first time—the optimal list size O(1/α), minimal batch complexity Ω(α⁻ᵟ), and estimation error O(α⁻ᵟ⁄²/√n). This breaks prior bottlenecks of O(1/α²) list size and O(α⁻¹⁄²/√n) error, while maintaining polynomial-time computability and significantly improving sample efficiency.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

We study the task of list-decodable linear regression using batches. A batch is called clean if it consists of i.i.d. samples from an unknown linear regression distribution. For a parameter $alpha in (0, 1/2)$, an unknown $alpha$-fraction of the batches are clean and no assumptions are made on the remaining ones. The goal is to output a small list of vectors at least one of which is close to the true regressor vector in $ell_2$-norm. [DJKS23] gave an efficient algorithm, under natural distributional assumptions, with the following guarantee. Assuming that the batch size $n$ satisfies $n geq ilde{Omega}(alpha^{-1})$ and the number of batches is $m = mathrm{poly}(d, n, 1/alpha)$, their algorithm runs in polynomial time and outputs a list of $O(1/alpha^2)$ vectors at least one of which is $ ilde{O}(alpha^{-1/2}/sqrt{n})$ close to the target regressor. Here we design a new polynomial time algorithm with significantly stronger guarantees under the assumption that the low-degree moments of the covariates distribution are Sum-of-Squares (SoS) certifiably bounded. Specifically, for any constant $delta>0$, as long as the batch size is $n geq Omega_{delta}(alpha^{-delta})$ and the degree-$Theta(1/delta)$ moments of the covariates are SoS certifiably bounded, our algorithm uses $m = mathrm{poly}((dn)^{1/delta}, 1/alpha)$ batches, runs in polynomial-time, and outputs an $O(1/alpha)$-sized list of vectors one of which is $O(alpha^{-delta/2}/sqrt{n})$ close to the target. That is, our algorithm achieves substantially smaller minimum batch size and final error, while achieving the optimal list size. Our approach uses higher-order moment information by carefully combining the SoS paradigm interleaved with an iterative method and a novel list pruning procedure. In the process, we give an SoS proof of the Marcinkiewicz-Zygmund inequality that may be of broader applicability.

Problem

• List-decodable linear regression with clean and corrupted batches.
• Output a small list of vectors close to the true regressor.
• Improved algorithm using higher-order moments and SoS bounds.

Innovation

• Uses higher-order moments for regression analysis
• Incorporates Sum-of-Squares certifiable bounds
• Implements novel list pruning procedure