Issues with Neural Tangent Kernel Approach to Neural Networks

2025-01-19

AI Summary

This paper investigates whether the Neural Tangent Kernel (NTK) accurately characterizes the actual training dynamics of deep neural networks, particularly how its predictive error scales with network depth. Method: We conduct rigorous theoretical re-derivation, full-batch gradient descent simulations, trajectory tracking of generalization error across multilayer networks, and systematic comparisons against Gaussian process kernels. Contribution/Results: We find that NTK kernel regression exhibits significant deviation from the true training trajectories in both optimization and generalization behavior; remarkably, a simple Gaussian kernel achieves comparable performance, indicating that the NTK fails to deliver its theoretical advantages in practice. This work provides the first empirical evidence that the NTK’s theoretical equivalence to infinite-width networks breaks down substantially under standard training settings, challenging its ability to model path-dependent optimization effects. Our findings establish critical empirical bounds on the practical applicability of the NTK framework.

Technology Category

• Machine Learning: Kernel Methods
• Natural Language Processing: Learning & Optimization for NLP
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks

Abstract

Neural tangent kernels (NTKs) have been proposed to study the behavior of trained neural networks from the perspective of Gaussian processes. An important result in this body of work is the theorem of equivalence between a trained neural network and kernel regression with the corresponding NTK. This theorem allows for an interpretation of neural networks as special cases of kernel regression. However, does this theorem of equivalence hold in practice? In this paper, we revisit the derivation of the NTK rigorously and conduct numerical experiments to evaluate this equivalence theorem. We observe that adding a layer to a neural network and the corresponding updated NTK do not yield matching changes in the predictor error. Furthermore, we observe that kernel regression with a Gaussian process kernel in the literature that does not account for neural network training produces prediction errors very close to that of kernel regression with NTKs. These observations suggest the equivalence theorem does not hold well in practice and puts into question whether neural tangent kernels adequately address the training process of neural networks.

Problem

• Neural Tangent Kernels
• Neural Network Training
• Depth Increase Prediction

Innovation

• Neural Tangent Kernels
• Gaussian Process Kernels
• Neural Network Layer Depth