Eigen-Spike Emergence and Quadratic Equivalents for Conjugate Kernels on Nonlinearly Separable Data

2026-05-28

AI Summary

This work addresses high-dimensional, nonlinearly separable data—such as the XOR problem—by establishing, for the first time, a robust quadratic-form deterministic equivalent for the neural network Conjugate Kernel (CK). Leveraging random matrix theory and BBP phase transition analysis, the study characterizes the asymptotic alignment between informative outlier eigenvalues (spikes) of the CK and the data labels. It precisely identifies the feasibility threshold for linear classification via CK features under varying sample complexity, signal-to-noise ratio, activation functions, and pre-trained feature transformations. The analysis rigorously delineates the emergence conditions of informative spikes, demonstrating that even in strongly nonlinear settings, the CK can effectively capture learnability through a linear model.

Abstract

Recent work in random matrix theory (RMT) has developed the notion of deterministic equivalents: typically linear surrogate models that approximate the spectral behavior of large nonlinear random matrices, such as nonlinear feature maps in neural networks (NNs). On the one hand, these deterministic equivalents make theoretical predictions tractable by reducing a complex model to a simpler model with properties that fall under the umbrella of classical RMT tools. However, this leaves open the question of whether this idealized linear equivalence remains meaningful when dealing with high-dimensional nonlinearly separable data, such as performing clssification on nonlinearly separable data. Motivated by this, we consider the conjugate kernel (CK), which is the nonlinear feature map of a feedforward NN, under a canonical nonlinearly separable dataset, the XOR problem; and we use the study of informative outlier eigenvalues in the CK and whether their corresponding eigenvectors asymptotically align with XOR labels as a proxy for nonlinear learnability. We develop a robust quadratic equivalent to the spiked CK matrix that enables a precise analysis of emergent informative spikes, as one modifies various knobs common in ML practice: sample complexity, signal-to-noise ratio (SNR), nonlinear activation choice, and pretrained features. In each of these scenarios, we derive a precise BBP-type phase transition in which linear classification via the CK eigenvectors becomes possible. Our analysis helps translate the power of deterministic equivalence tools in RMT to study problems of practical relevance in ML.

Problem

• nonlinearly separable data
• conjugate kernel
• deterministic equivalents
• informative eigenvalues
• random matrix theory

Innovation

• quadratic equivalent
• conjugate kernel
• informative eigen-spike
• nonlinear separability
• BBP phase transition