Problem
Research questions and friction points this paper is trying to address.
Naive Bayes
smoothing
high-cardinality data
calibration
class imbalance
Hierarchical Empirical-Bayes Naive Bayes: Minimax Smoothing and Calibration with AODE Extension
🤖 AI Summary
This work addresses the significant bias introduced by fixed smoothing in traditional Naive Bayes classifiers under high-cardinality, small-sample, or class-imbalanced settings. The authors propose Hierarchical Empirical Bayes Naive Bayes (HEB-NB), which adaptively learns the concentration parameters of Dirichlet priors via type-II maximum likelihood, enabling cross-class information sharing while preserving closed-form inference. The approach is further extended to AODE, yielding HEB-AODE. For the first time, adaptive smoothing is integrated with minimax theory to establish non-asymptotic ℓ₁ error bounds, and the method is proven to achieve strictly lower finite-sample risk than Laplace smoothing. Evaluated on 31 UCI/OpenML benchmarks, HEB-NB attains the best average Friedman rank, reducing log loss by up to 22.1% on high-cardinality datasets; HEB-AODE consistently outperforms standard AODE, and when combined with mutual information weighting, reduces Top-1 Expected Calibration Error (ECE) by 41%–70%.
📝 Abstract
The Naive Bayes (NB) classifier remains a standard choice for categorical data, yet its widely used smoothing rules, such as Laplace, Lidstone, Krichevsky-Trofimov, and the $m$-estimate, all prescribe a fixed smoothing strength that ignores feature cardinality, sample size, and class imbalance, inducing a non-vanishing bias on modern high-cardinality tabular data. We propose hierarchical empirical-Bayes Naive Bayes (HEB-NB), in which each class-feature conditional probability is smoothed by a Dirichlet prior whose concentration is learned data-adaptively via Type-II maximum likelihood, enabling principled information sharing across classes while retaining closed-form inference. We further introduce HEB average one-dependence estimators (HEB-AODE), showing that the adaptive smoothing transfers cleanly to structural relaxations of NB. Theoretically, we establish a non-asymptotic $\ell_1$ error bound for HEB-NB matching the empirical-distribution minimax rate plus a vanishing data-adaptive bias, together with a matching Laplace-tight lower bound that yields a finite-sample, risk-level strict separation from Laplace. We further derive a plug-in excess Bayes-risk bound via total-variation tensorization and a population top-1 expected calibration error (ECE) corollary. Empirically, across 31 UCI and OpenML benchmarks, HEB-NB attains the best average Friedman rank on probabilistic metrics, with up to 22.1% log-loss reductions on high-cardinality datasets and consistent improvements of HEB-AODE over vanilla AODE. Combining HEB-NB with mutual-information weighting reduces top-1 ECE by 41%-70%, demonstrating substantial gains in probabilistic accuracy and calibration.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Hierarchical Empirical-Bayes
Adaptive Smoothing
Naive Bayes
Minimax Calibration
AODE Extension
🔎 Similar Papers
No similar papers found.
