On Kernel Eigen-alignments of KRR: Reconstruction and Generalization

2026-05-14

AI Summary

This work investigates how the alignment between eigenvectors of kernel matrices and learning targets influences generalization performance in finite-sample settings within kernel methods. Framed in kernel ridge regression, the study leverages matrix perturbation theory and spectral analysis to establish, for the first time, finite-sample generalization error bounds that jointly account for feature alignment, eigenvalue magnitude, and spectral gap. The theoretical analysis reveals that strong generalization arises when there is high alignment, large eigenvalues, or a pronounced spectral gap. Notably, under high-rank kernels, low reconstruction error proves insufficient for predicting generalization capability, thereby exposing a fundamental limitation of conventional reconstruction-based metrics.

Abstract

This paper investigates the critical role of eigenalignments between the kernel matrix and learning targets in achieving robust generalization in learning problems. We establish a direct connection between generalization performance in kernel methods and the estimation of eigenvectors and eigenvalues of matrices, offering a more intuitive understanding compared to prior work with minimal assumptions. We also show that, since the prediction task in KRR is essentially the weighted sum of eigenvectors/singular vectors, by analyzing how much error can be caused by perturbations to the kernel matrix, we can then derive a bound on this generalization error using the estimation stability of matrix eigenvalues and eigenvectors. Compared with previous work, our analysis concentrates on finite-sample settings and on the generalization error arising from having a suboptimal finite training set. Our findings reveal that in kernel methods, as long as the kernel is of high rank, the near-zero reconstruction error can be trivially obtained, implying that the reconstruction error will have limited predictive power for generalization. Finally, we establish a generalization bound from an eigenvalues/eigenvectors estimation perspective, showing that strong generalization requires increasing eigenvector alignment, eigenvalue magnitude, or gaps between consecutive eigenvalues.

Problem

• kernel methods
• generalization error
• eigen-alignment
• KRR
• finite-sample analysis

Innovation

• eigenalignment
• kernel ridge regression
• generalization bound
• eigenvector estimation
• finite-sample analysis