Adaptive Kernel Density Estimation with Pre-training

2026-05-13

AI Summary

This work addresses the efficiency and accuracy limitations of high-dimensional kernel density estimation, which stem from the difficulty of designing location-adaptive kernels. It introduces, for the first time, a pretraining paradigm into nonparametric density estimation by leveraging a pretrained neural network to data-dependently recommend adaptive kernel functions for each sample. To handle distributional shifts between pretraining and target domains, the method incorporates a fine-tuning strategy. The proposed approach substantially improves high-dimensional density estimation performance: when the target distribution is close to the pretraining distribution, it achieves markedly higher accuracy than conventional methods; even under significant distributional discrepancies, fine-tuning effectively restores much of the performance gain.

Abstract

Density estimation in high-dimensional settings is an important and challenging statistical problem.Traditional methods based on kernel smoothing are inefficient in high dimensions due to the difficulties in specifying appropriate location-adaptive kernels. In this work, we introduce pre-training, a key idea behind many cutting-edge AI technologies, to the context of non-parametric density estimation. By establishing a pre-trained neural network that can recommend an appropriate location-adaptive kernel for each sample point, efficient density estimation with adaptive kernels is achieved in high dimensions. A wide range of numerical experiments show that this strategy is highly effective for improving density-estimation accuracy, when the target distribution is close to the distribution family for pre-training. When the target distribution is substantially different from the pre-training distribution family, the benefit from the proposed pre-training strategy may be diluted, but can be reactivated by an additional fine-tuning procedure.

Problem

• density estimation
• high-dimensional
• kernel density estimation
• adaptive kernels
• non-parametric

Innovation

• pre-training
• adaptive kernel density estimation
• high-dimensional density estimation
• neural network
• fine-tuning