Estimating the Joint Distribution of Two Binary Variables with Marginal Statistics

2025-05-06

AI Summary

In clinical trial simulations, privacy constraints and limited data availability often restrict access to only the marginal frequencies of two binary variables—lacking individual-level data and association information—thus impeding accurate estimation of the joint distribution. To address this, we propose the first purely marginal maximum likelihood estimation framework: given univariate frequency counts from heterogeneous sample sizes, it reconstructs the exact 2×2 joint distribution without requiring covariates, prior assumptions about association, or disclosure of individual records. Our method solves a numerically optimized likelihood problem subject to marginal constraints using the L-BFGS algorithm. Extensive evaluations across diverse simulation scenarios and real-world pharmaceutical trial datasets demonstrate an average joint probability estimation error <0.01, confirming both high accuracy and strong robustness. This advancement significantly enhances the fidelity of simulated trial data and improves the reliability of downstream drug development decisions.

Technology Category

• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Clinical trial simulation (CTS) is critical in new drug development, providing insight into safety and efficacy while guiding trial design. Achieving realistic outcomes in CTS requires an accurately estimated joint distribution of the underlying variables. However, privacy concerns and data availability issues often restrict researchers to marginal summary-level data of each variable, making it challenging to estimate the joint distribution due to the lack of access to individual-level data or relational summaries between variables. We propose a novel approach based on the method of maximum likelihood that estimates the joint distribution of two binary variables using only marginal summary data. By leveraging numerical optimization and accommodating varying sample sizes across studies, our method preserves privacy while bypassing the need for granular or relational data. Through an extensive simulation study covering a diverse range of scenarios and an application to a real-world dataset, we demonstrate the accuracy, robustness, and practicality of our method. This method enhances the generation of realistic simulated data, thereby improving decision-making processes in drug development.

Problem

• Estimating joint distribution of binary variables using marginal data
• Addressing privacy constraints in clinical trial simulations
• Enhancing drug development with accurate distribution estimates

Innovation

• Maximum likelihood method for joint distribution estimation
• Numerical optimization with varying sample sizes
• Privacy-preserving marginal summary data usage