Why High-rank Neural Networks Generalize?: An Algebraic Framework with RKHSs

2025-09-26

AI Summary

Why do deep neural networks with high-rank weight matrices exhibit strong generalization performance, despite conventional learning-theoretic bounds failing under such settings? Method: We develop an algebraic analytical framework integrating Koopman operators, group representation theory, and reproducing kernel Hilbert spaces (RKHS), mapping neural network weight structures to interpretable algebraic representations under group actions, and designing novel kernels tailored to high-rank architectures. Contribution/Results: Within this framework, we derive the first Rademacher complexity upper bound applicable to general high-rank networks—breaking reliance on restrictive low-rank or linearization assumptions. The bound explicitly characterizes the interplay among weight matrix rank, symmetry properties, and generalization error, substantially enhancing theoretical interpretability and practical applicability. Moreover, our work establishes the first substantive application of Koopman operator theory to generalization analysis in deep learning.

Technology Category

• Machine Learning: Kernel Methods
• Computer Vision: Representation Learning for Vision
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We derive a new Rademacher complexity bound for deep neural networks using Koopman operators, group representations, and reproducing kernel Hilbert spaces (RKHSs). The proposed bound describes why the models with high-rank weight matrices generalize well. Although there are existing bounds that attempt to describe this phenomenon, these existing bounds can be applied to limited types of models. We introduce an algebraic representation of neural networks and a kernel function to construct an RKHS to derive a bound for a wider range of realistic models. This work paves the way for the Koopman-based theory for Rademacher complexity bounds to be valid for more practical situations.

Problem

• Explaining why high-rank neural networks generalize well in practice
• Developing new Rademacher complexity bounds using Koopman operators
• Extending generalization theory to wider range of practical models

Innovation

• Uses Koopman operators for complexity bounds
• Applies group representations to neural networks
• Constructs RKHS with algebraic network representation