The Poisson tensor completion non-parametric differential entropy estimator

2025-05-08

AI Summary

This paper addresses the challenge of nonparametric estimation of differential entropy for high-dimensional continuous distributions, particularly when histogram bins are sparse and sample sizes are small—scenarios where classical estimators suffer from substantial bias. We propose a Poisson tensor completion entropy estimator grounded in spatial Poisson process modeling. Specifically, we explicitly model the histogram as an anisotropic spatial Poisson process and perform global intensity measure completion via low-rank Poisson tensor decomposition. This ensures nonnegative, statistically consistent density estimates. By integrating nonparametric density estimation with a low-rank tensor structural prior, our method achieves significantly improved entropy estimation accuracy and robustness over conventional histogram-based estimators—especially under sub-Gaussian distributions—effectively mitigating bias arising from low-count bins.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Reasoning under Uncertainty: Graphical Models
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

We introduce the Poisson tensor completion (PTC) estimator, a non-parametric differential entropy estimator. The PTC estimator leverages inter-sample relationships to compute a low-rank Poisson tensor decomposition of the frequency histogram. Our crucial observation is that the histogram bins are an instance of a space partitioning of counts and thus can be identified with a spatial Poisson process. The Poisson tensor decomposition leads to a completion of the intensity measure over all bins -- including those containing few to no samples -- and leads to our proposed PTC differential entropy estimator. A Poisson tensor decomposition models the underlying distribution of the count data and guarantees non-negative estimated values and so can be safely used directly in entropy estimation. We believe our estimator is the first tensor-based estimator that exploits the underlying spatial Poisson process related to the histogram explicitly when estimating the probability density with low-rank tensor decompositions or tensor completion. Furthermore, we demonstrate that our PTC estimator is a substantial improvement over standard histogram-based estimators for sub-Gaussian probability distributions because of the concentration of norm phenomenon.

Problem

• Estimates differential entropy using Poisson tensor completion
• Improves histogram-based entropy estimation for sub-Gaussian distributions
• Models count data distribution via low-rank tensor decomposition

Innovation

• Low-rank Poisson tensor decomposition for entropy estimation
• Spatial Poisson process modeling of histogram bins
• Non-negative tensor completion for density estimation