Wilsonian Renormalization of Neural Network Gaussian Processes

2024-05-09

Machine Learning: Science and Technology

AI Summary

This work addresses the lack of a universal theoretical framework for understanding feature learning in deep neural networks (DNNs). We introduce the Wilson renormalization group (RG) formalism into Gaussian process (GP) regression for the first time. By constructing a continuous RG flow in kernel function space—where data scale serves as the infrared cutoff—we systematically integrate out non-learnable high-frequency kernel modes, thereby naturally separating learnable from non-learnable features. Our theoretical analysis reveals the universal scaling behavior of the ridge parameter under the RG flow and characterizes the input dependence of the flow equation for non-Gaussian inputs. Crucially, we establish a rigorous correspondence between the RG flow and kernel learnability, yielding the first analytically tractable renormalization-theoretic framework for DNN feature learning. This framework enables principled identification and classification of potential universality classes in kernel learning and neural representation.

Technology Category

• Machine Learning: Deep Learning Theory
• Natural Language Processing: Learning & Optimization for NLP
• Computer Vision: Representation Learning for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Separating relevant and irrelevant information is key to any modeling process or scientific inquiry. Theoretical physics offers a powerful tool for achieving this in the form of the renormalization group (RG). Here we demonstrate a practical approach to performing Wilsonian RG in the context of Gaussian Process (GP) Regression. We systematically integrate out the unlearnable modes of the GP kernel, thereby obtaining an RG flow of the GP in which the data sets the IR scale. In simple cases, this results in a universal flow of the ridge parameter, which becomes input-dependent in the richer scenario in which non-Gaussianities are included. In addition to being analytically tractable, this approach goes beyond structural analogies between RG and neural networks by providing a natural connection between RG flow and learnable vs. unlearnable modes. Studying such flows may improve our understanding of feature learning in deep neural networks, and enable us to identify potential universality classes in these models.

Problem

• Separating relevant and irrelevant information in modeling using renormalization group
• Performing Wilsonian RG in Gaussian Process Regression systematically
• Understanding feature learning in deep neural networks via RG flow

Innovation

• Wilsonian RG applied to Gaussian Process Regression
• Systematically integrate out unlearnable GP kernel modes
• RG flow connects to learnable vs unlearnable modes