Demystifying Spectral Bias on Real-World Data

2024-06-04

AI Summary

Kernel ridge regression (KRR) and Gaussian processes (GPs) suffer from degraded learnability on real-world data due to rapid eigenvalue decay of the kernel matrix, impairing their ability to recover target functions. Method: We propose a spectral learnability analysis framework that introduces the idealized data measure’s eigen-spectrum as a theoretical benchmark, quantifying spectral deviation of real data from this benchmark. Leveraging symmetry properties of the kernel operator, we derive tight theoretical bounds on learnability, revealing bottlenecks in learning components aligned with high-eigenvalue eigenvectors. The framework integrates kernel spectral analysis, eigenvalue estimation, and symmetry-driven generalization theory to quantitatively characterize the learnability of kernel methods under realistic data distributions. Results: Experiments demonstrate substantial improvements in predictive accuracy for learnability estimation. The framework establishes a novel paradigm for theoretically interpreting and practically deploying kernel methods on non-ideal, real-world data.

Technology Category

• Machine Learning: Kernel Methods
• Game Theory and Economic Paradigms: Adversarial Learning
• Knowledge Representation and Reasoning: Knowledge Acquisition

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Kernel ridge regression (KRR) and Gaussian processes (GPs) are fundamental tools in statistics and machine learning, with recent applications to highly over-parameterized deep neural networks. The ability of these tools to learn a target function is directly related to the eigenvalues of their kernel sampled on the input data distribution. Targets that have support on higher eigenvalues are more learnable. However, solving such eigenvalue problems on real-world data remains a challenge. Here, we consider cross-dataset learnability and show that one may use eigenvalues and eigenfunctions associated with highly idealized data measures to reveal spectral bias on complex datasets and bound learnability on real-world data. This allows us to leverage various symmetries that realistic kernels manifest to unravel their spectral bias.

Problem

• Eigenvalue challenges in real-world data
• Cross-dataset learnability analysis
• Spectral bias in complex datasets

Innovation

• Kernel ridge regression analysis
• Gaussian processes application
• Cross-dataset learnability exploration