A Function-Space Dichotomy for Compositional Learning: Exponential Sub-Optimality of the Neural Tangent Kernel

2026-07-07

AI Summary

This work elucidates the mechanism by which deep ReLU networks substantially outperform Neural Tangent Kernels (NTK) on compositional tasks, establishing the first quantitative theoretical framework that characterizes their performance gap. By introducing a dichotomy between Fourier complexity and architectural complexity, the analysis examines expressivity differences in function spaces over the unit circle, demonstrating that this advantage stems from the intrinsic structure of the target function rather than being a universal phenomenon. Theoretically, for an iterated sawtooth function of depth \(L\), NTK requires \(\Omega(4^L)\) samples to learn, whereas deep networks succeed with only polynomial sample complexity. Empirical validation on sparse parity models confirms that neural networks achieve test errors 4–6 orders of magnitude lower than those of NTK.

Abstract

A persistent empirical observation is that trained neural networks outperform their neural tangent kernel (NTK) limit on tasks with compositional structure, yet a quantitative account of $\textbf{when}$ and $\textbf{by how much}$ has been lacking. Working on the unit circle, we give such an account through a dichotomy between two complexity measures of the target: its $\textbf{Fourier complexity}$, which controls NTK kernel regression, and its $\textbf{architectural complexity}$, which controls learning over depth-$L$, width-$w$ ReLU networks with the variation norm of the weights bounded by $R$. We first characterize the minimax rate of the architecture class $\mathcal{C}_{L,w,R}$, pinning it down up to a single factor of $L$: between $Ω(Lw^2R^2/n)$ and $\tilde{O}(L^2w^2R^2/n)$. We then show the NTK estimator sits $\textbf{exponentially}$ above this floor whenever the two complexities decouple: for the depth-$L$ iterated sawtooth, NTK regression needs $Ω(4^L)$ samples while the minimax floor is polynomial in $L$. Numerical experiments confirm the theoretical claims: on bandlimited smooth targets, the NTK is competitive or better, while on the hypercube sparse-parity model, a standard two-layer network beats the NTK by four to six orders of magnitude in test error. The gap is thus a function-space property, a mismatch between the kernel's smoothness bias and the target's compositional structure, rather than a generic kernel-versus-network phenomenon.

Problem

• compositional learning
• neural tangent kernel
• function-space dichotomy
• Fourier complexity
• architectural complexity

Innovation

• compositional learning
• neural tangent kernel
• Fourier complexity
• architectural complexity
• minimax rate