Learning Smooth Populations of Parameters with Trial Heterogeneity

2025-07-30

AI Summary

This paper addresses nonparametric estimation of the mixing distribution in binomial mixture models, focusing on the practically relevant yet underexplored setting where trial parameters are heterogeneous and the mixing density is s-smooth. We propose a novel kernel-based density estimation method and, for the first time under heterogeneous trials, derive a fast convergence rate dependent on the harmonic mean of trial sizes; its homogeneous special case improves upon existing optimal rates. Additionally, we construct a nonparametric estimator for the difference between two mixing densities, leveraging the potentially higher-order smoothness of the difference function relative to individual densities. The methodology applies to both i.i.d. and binomial mixture data. An empirical analysis of Pennsylvania criminal justice data reveals that defendants assigned private counsel face systematically higher conviction rates than those represented by public defenders—a disparity likely attributable to confounding by case severity.

Technology Category

• Machine Learning: Kernel Methods
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We consider the classical problem of estimating the mixing distribution of binomial mixtures, but under trial heterogeneity and smoothness. This problem has been studied extensively when the trial parameter is homogeneous, but not under the more general scenario of heterogeneous trials, and only within a low smoothness regime, where the resulting rates are slow. Under the assumption that the density is s-smooth, we derive fast error rates for the kernel density estimator under trial heterogeneity that depend on the harmonic mean of the trials. Importantly, even when reduced to the homogeneous case, our result improves on the state-of-the-art rate of Ye and Bickel (2021). We also study nonparametric estimation of the difference between two densities, which can be smoother than the individual densities, in both i.i.d. and binomial-mixture settings. Our work is motivated by an application in criminal justice: comparing conviction rates of indigent representation in Pennsylvania. We find that the estimated conviction rates for appointed counsel (court-appointed private attorneys) are generally higher than those for public defenders, potentially due to a confounding factor: appointed counsel are more likely to take on severe cases.

Problem

• Estimating binomial mixtures with trial heterogeneity and smoothness
• Improving error rates for kernel density estimation in heterogeneous trials
• Comparing densities in binomial-mixture settings for criminal justice analysis

Innovation

• Kernel density estimator for heterogeneous trials
• Fast error rates with harmonic mean
• Nonparametric estimation of density differences