Kernel ridge regression under power-law data: spectrum and generalization

2025-10-06

AI Summary

This paper investigates the spectral properties and generalization performance of kernel ridge regression (KRR) under high-dimensional anisotropic power-law covariance structures. Unlike conventional settings assuming a power-law decay of the kernel’s eigenvalue spectrum, we consider the more realistic scenario where the *data covariance itself* follows a power-law decay. We establish, for the first time, a rigorous spectral inheritance mechanism for polynomial inner-product kernels: data anisotropy governs the kernel’s eigenvalue decay rate, and the effective dimension—not the ambient dimension—dictates sample complexity. Leveraging high-dimensional asymptotics and random matrix theory within an anisotropic Gaussian data model, we prove that KRR achieves strictly superior generalization error on such structured data, with sample complexity substantially lower than in the isotropic case. Our results fundamentally reveal the advantage of nonlinear learning when data reside on implicit low-dimensional structures.

Technology Category

• Machine Learning: Kernel Methods
• Data Mining & Knowledge Management: Data Compression
• Search and Optimization: Non-convex Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

In this work, we investigate high-dimensional kernel ridge regression (KRR) on i.i.d. Gaussian data with anisotropic power-law covariance. This setting differs fundamentally from the classical source & capacity conditions for KRR, where power-law assumptions are typically imposed on the kernel eigen-spectrum itself. Our contributions are twofold. First, we derive an explicit characterization of the kernel spectrum for polynomial inner-product kernels, giving a precise description of how the kernel eigen-spectrum inherits the data decay. Second, we provide an asymptotic analysis of the excess risk in the high-dimensional regime for a particular kernel with this spectral behavior, showing that the sample complexity is governed by the effective dimension of the data rather than the ambient dimension. These results establish a fundamental advantage of learning with power-law anisotropic data over isotropic data. To our knowledge, this is the first rigorous treatment of non-linear KRR under power-law data.

Problem

• Analyzing kernel ridge regression with anisotropic power-law data covariance
• Characterizing kernel spectrum inheritance from data decay patterns
• Establishing sample complexity advantage using data effective dimension

Innovation

• Characterized kernel spectrum for polynomial inner-product kernels
• Analyzed excess risk in high-dimensional regime asymptotically
• Established sample complexity governed by data effective dimension