A Practical Introduction to Kernel Discrepancies: MMD, HSIC&KSD

2025-03-04

AI Summary

This work addresses the kernel selection and bandwidth sensitivity challenges inherent in kernel-based discrepancy measures—specifically Maximum Mean Discrepancy (MMD), Hilbert–Schmidt Independence Criterion (HSIC), and Kernel Stein Discrepancy (KSD)—for distribution comparison, independence testing, and generative model evaluation. We propose a unified computational framework and a multi-kernel adaptive fusion estimator grounded in Hilbert space embeddings and Stein operator theory. Our method integrates V- and U-statistics, employs efficient incomplete U-statistic approximations, and incorporates a data-driven bandwidth adaptation strategy. Compared to single-kernel approaches, the proposed estimator substantially improves statistical power in small-sample and high-dimensional settings, while ensuring reproducibility and ease of hyperparameter tuning. The resulting toolkit provides a theoretically coherent and practically accessible unified implementation for all three major kernel discrepancies.

Technology Category

• Machine Learning: Kernel Methods
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

This article provides a practical introduction to kernel discrepancies, focusing on the Maximum Mean Discrepancy (MMD), the Hilbert-Schmidt Independence Criterion (HSIC), and the Kernel Stein Discrepancy (KSD). Various estimators for these discrepancies are presented, including the commonly-used V-statistics and U-statistics, as well as several forms of the more computationally-efficient incomplete U-statistics. The importance of the choice of kernel bandwidth is stressed, showing how it affects the behaviour of the discrepancy estimation. Adaptive estimators are introduced, which combine multiple estimators with various kernels, addressing the problem of kernel selection.

Problem

• Introduces kernel discrepancies: MMD, HSIC, KSD.
• Presents estimators like V-statistics, U-statistics.
• Addresses kernel selection and bandwidth impact.

Innovation

• Introduces MMD, HSIC, and KSD kernel discrepancies.
• Uses V-statistics and U-statistics for discrepancy estimation.
• Proposes adaptive estimators for kernel selection optimization.