SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model

2026-06-29

AI Summary

This work investigates the dynamics of two-layer ReLU neural networks trained via online mini-batch SGD on high-dimensional Boolean hypercubes, where the target signal is an XOR function confounded with linear spurious features. While neural networks are known to exploit such spurious correlations—often termed shortcut features—a rigorous theoretical understanding of this phenomenon has been lacking. The paper provides the first end-to-end theoretical characterization of spurious feature learning, revealing a sharp phase transition in optimization dynamics: SGD exponentially favors strong spurious features, completely suppressing the XOR signal even when the sample complexity theoretically permits its recovery. In regimes with weaker spurious correlations, the network eventually learns the XOR function but does not unlearn the shortcut, highlighting a persistent entanglement between genuine and spurious learning pathways.

Abstract

Neural networks are known to be susceptible to over-reliance on spurious correlations. However, the precise mechanism by which models exploit shortcut features is not fully understood, and algorithms to mitigate this behavior rely on as yet unjustified assumptions about the learned representations. In this work, we provide the first end-to-end theoretical characterization of spurious feature learning for two-layer ReLU neural networks trained by online minibatch SGD on the logistic loss. We consider data drawn from the high-dimensional Boolean hypercube with a quadratic signal function (namely XOR) and a linear spurious correlation. We show that SGD learns the spurious feature first, and exponentially fast. Moreover, the optimization dynamics couple the spurious and signal features, with a stronger spurious component inhibiting signal feature learning. Our analysis reveals precise phase transitions in the learning dynamics. In the first phase, alignment between the signs of the spurious feature and second-layer weight drives rapid growth of the spurious feature. In the second phase, large majority group margin slows learning and the signal feature remains suppressed. When the spurious correlation is maximally strong, we show theoretically that the spurious feature dominates even at the sample complexity threshold where XOR would be learned in isolation (i.e., if the spurious feature was absent). In contrast, when the correlation strength is constant, we provide preliminary empirical evidence that the model can eventually learn the XOR signal, although the spurious feature is not forgotten.

Problem

• spurious correlation
• shortcut learning
• XOR model
• neural network optimization
• feature suppression

Innovation

• spurious correlation
• shortcut learning
• SGD dynamics
• XOR model
• theoretical analysis