Statistical Query Hardness of Multiclass Linear Classification with Random Classification Noise

2025-02-17

AI Summary

This work investigates the statistical query (SQ) learnability of multiclass linear classification (MLC) under the distribution-independent PAC model with random classification noise (RCN). We consider label generation by a multiclass linear classifier corrupted by RCN. Our key finding is the first demonstration of an intrinsic computational separation: for ≥3 classes, any SQ algorithm requires superpolynomial query complexity to achieve nontrivial 0–1 error, whereas the binary case admits a polynomial-time solution. Technically, we introduce a separation-degree-driven lower-bound framework based on noise matrix modeling and geometric characterization of decision boundaries. We rigorously prove a superpolynomial SQ lower bound under constant separation for three classes; for more classes or smaller separation, no SQ algorithm can surpass the random-guess baseline. This establishes the first computational hardness theory for MLC under RCN, challenging the prevailing assumption that efficient binary-classification algorithms generalize naturally to the multiclass setting.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

We study the task of Multiclass Linear Classification (MLC) in the distribution-free PAC model with Random Classification Noise (RCN). Specifically, the learner is given a set of labeled examples $(x, y)$, where $x$ is drawn from an unknown distribution on $R^d$ and the labels are generated by a multiclass linear classifier corrupted with RCN. That is, the label $y$ is flipped from $i$ to $j$ with probability $H_{ij}$ according to a known noise matrix $H$ with non-negative separation $sigma: = min_{i eq j} H_{ii}-H_{ij}$. The goal is to compute a hypothesis with small 0-1 error. For the special case of two labels, prior work has given polynomial-time algorithms achieving the optimal error. Surprisingly, little is known about the complexity of this task even for three labels. As our main contribution, we show that the complexity of MLC with RCN becomes drastically different in the presence of three or more labels. Specifically, we prove super-polynomial Statistical Query (SQ) lower bounds for this problem. In more detail, even for three labels and constant separation, we give a super-polynomial lower bound on the complexity of any SQ algorithm achieving optimal error. For a larger number of labels and smaller separation, we show a super-polynomial SQ lower bound even for the weaker goal of achieving any constant factor approximation to the optimal loss or even beating the trivial hypothesis.

Problem

• Multiclass Linear Classification complexity
• Random Classification Noise impact
• Statistical Query lower bounds

Innovation

• Multiclass Linear Classification
• Random Classification Noise
• Statistical Query lower bounds