Bayes Error Rate Estimation in Difficult Situations

2025-05-21

arXiv.org

AI Summary

This study addresses the accurate estimation of the Bayes error rate (BER) in multiclass classification under small-sample, high-dimensional, and class-distribution-agnostic settings. Using large-scale Monte Carlo simulations, we systematically establish the first quantitative confidence-bound benchmarks for nonparametric BER estimators, evaluating the precision limits of k-nearest neighbors (kNN), generalized Henze–Penrose divergence, and kernel density estimation. Key results: kNN achieves a 95% confidence interval width <5% with only 1,000 samples per class in ≤3 dimensions, but requires 2,500 samples per class in 4 dimensions—revealing a critical dimensional dependence on sample complexity. All other methods fail to meet practical accuracy thresholds. This work provides the first empirically validated precision benchmark for BER estimation and offers dimension-aware guidance for estimator selection and sample sizing.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

The Bayes Error Rate (BER) is the fundamental limit on the achievable generalizable classification accuracy of any machine learning model due to inherent uncertainty within the data. BER estimators offer insight into the difficulty of any classification problem and set expectations for optimal classification performance. In order to be useful, the estimators must also be accurate with a limited number of samples on multivariate problems with unknown class distributions. To determine which estimators meet the minimum requirements for"usefulness", an in-depth examination of their accuracy is conducted using Monte Carlo simulations with synthetic data in order to obtain their confidence bounds for binary classification. To examine the usability of the estimators on real-world applications, new test scenarios are introduced upon which 2500 Monte Carlo simulations per scenario are run over a wide range of BER values. In a comparison of k-Nearest Neighbor (kNN), Generalized Henze-Penrose (GHP) divergence and Kernel Density Estimation (KDE) techniques, results show that kNN is overwhelmingly the more accurate non-parametric estimator. In order to reach the target of an under 5 percent range for the 95 percent confidence bounds, the minimum number of required samples per class is 1000. As more features are added, more samples are needed, so that 2500 samples per class are required at only 4 features. Other estimators do become more accurate than kNN as more features are added, but continuously fail to meet the target range.

Problem

• Estimating Bayes Error Rate with limited samples
• Evaluating estimator accuracy in multivariate classification
• Determining minimum sample size for reliable BER estimation

Innovation

• Monte Carlo simulations for confidence bounds
• k-Nearest Neighbor as accurate non-parametric estimator
• Minimum 1000 samples per class requirement