Loop Corrections to the Training and Generalization Errors of Random Feature Models

2026-04-14

AI Summary

This work addresses the limitations of conventional mean-field kernel approximations in accurately capturing the training and generalization errors of finite-width random feature models. From a statistical physics perspective, it introduces— for the first time—loop corrections from effective field theory into this setting, systematically accounting for higher-order effects induced by fluctuations of the kernel function. By integrating tools from random matrix theory with numerical experiments, the study derives explicit finite-width correction formulas for training, test, and generalization errors, precisely characterizing their scaling behavior with respect to network width. The analysis further validates the significance of higher-order statistical contributions that go beyond the mean-field approximation, offering a more refined theoretical understanding of finite-size effects in overparameterized models.

Technology Category

• Machine Learning: Kernel Methods
• Cognitive Modeling & Cognitive Systems: Other Foundations of Cognitive Modeling & Systems
• Reasoning under Uncertainty: Graphical Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

We investigate random feature models in which neural networks sampled from a prescribed initialization ensemble are frozen and used as random features, with only the readout weights optimized. Adopting a statistical-physics viewpoint, we study the training, test, and generalization errors beyond the mean-kernel approximation. Since the predictor is a nonlinear functional of the induced random kernel, the ensemble-averaged errors depend not only on the mean kernel but also on higher-order fluctuation statistics. Within an effective field-theoretic framework, these finite-width contributions naturally appear as loop corrections. We derive the loop corrections to the training, test, and generalization errors, obtain their scaling laws, and support the theory with experimental verification.

Problem

• random feature models
• generalization error
• training error
• loop corrections
• finite-width effects

Innovation

• loop corrections
• random feature models
• finite-width effects
• statistical physics
• generalization error