Kernel Density Machines

2025-04-30

AI Summary

This paper addresses density ratio estimation for general probability measures on countably generated measurable spaces, removing conventional assumptions of Lebesgue density existence or continuity. We propose the Kernel Density Machine (KDM), a novel density ratio estimator grounded in reproducing kernel Hilbert spaces (RKHS), which achieves provably asymptotically consistent estimation without any density assumptions—its first such guarantee. To ensure both theoretical rigor and scalability to large datasets, KDM employs a controlled-error low-rank approximation of the kernel matrix. Theoretical analysis establishes a functional central limit theorem and finite-sample error bounds. Empirical evaluations demonstrate that KDM significantly improves estimation accuracy and robustness over existing methods on both synthetic and real-world datasets.

Technology Category

• Machine Learning: Kernel Methods
• Reasoning under Uncertainty: Relational Probabilistic Models
• Intelligent Robots: State Estimation

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We introduce kernel density machines (KDM), a novel density ratio estimator in a reproducing kernel Hilbert space setting. KDM applies to general probability measures on countably generated measurable spaces without restrictive assumptions on continuity, or the existence of a Lebesgue density. For computational efficiency, we incorporate a low-rank approximation with precisely controlled error that grants scalability to large-sample settings. We provide rigorous theoretical guarantees, including asymptotic consistency, a functional central limit theorem, and finite-sample error bounds, establishing a strong foundation for practical use. Empirical results based on simulated and real data demonstrate the efficacy and precision of KDM.

Problem

• Estimates density ratios without restrictive assumptions
• Incorporates low-rank approximation for scalability
• Provides theoretical guarantees and empirical validation

Innovation

• Kernel density machines estimate density ratios efficiently
• Low-rank approximation ensures scalability in large samples
• Theoretical guarantees include consistency and error bounds