A Rigorous, Tractable Measure of Model Complexity

2026-05-20

AI Summary

This work proposes a unified measure of model complexity based on gradient similarity under input perturbations, applicable to both parametric and non-parametric models. Existing complexity metrics often struggle to balance theoretical rigor with computational efficiency; in contrast, the proposed measure achieves both while offering a coherent framework that subsumes classical notions such as polynomial degree and kernel lengthscale. The approach provides a novel interpretation of the double descent phenomenon by revealing how model complexity evolves throughout the training process. Through rigorous theoretical analysis and empirical validation across diverse model classes—including neural networks and random Fourier features—the method demonstrates strong mathematical grounding and practical scalability, effectively capturing the nuanced dynamics of complexity in modern machine learning systems.

Abstract

An accurate assessment of a model's complexity is crucial for topics such as interpretation, generalization, and model selection. However, most existing complexity measures either rely on heuristic assumptions or are computationally prohibitive. In this paper, we present a mathematically rigorous yet easy-to-compute measure of model complexity that is based on the similarities between the model gradients across inputs. It is thus well-defined for any parametric model, but also for kernel-based non-parametric models. We prove that our measure of complexity generalizes model-specific complexity measures such as polynomial degree (for polynomial regression), kernel length scale (for Matérn kernels), number of neighbors (for k-nearest neighbors), number of splits (for decision trees), and number of trees (for random forests). We also use our measure to obtain new insights into the double descent phenomenon for random Fourier features, random forests, neural networks, and gradient boosting.

Problem

• model complexity
• complexity measure
• generalization
• double descent
• parametric models

Innovation

• model complexity
• gradient similarity
• double descent
• non-parametric models
• tractable measure