Integration of Individual Participant and Aggregate Data Under Dataset Shift: Summary Statistic Comparison and Scalable Computation

2026-03-01

AI Summary

This study addresses the efficient integration of individual participant data (IPD) and aggregate data (AD) under dataset shift, systematically evaluating the impact of different AD formats on estimation efficiency within a constrained maximum likelihood framework. The authors propose a fast, non-iterative estimation algorithm robust to both covariate shift and prior probability shift. Their analysis reveals that outcome-stratified summary statistics—such as case/control counts—substantially outperform covariate-stratified summaries, particularly in settings with continuous outcomes, yielding markedly improved estimation efficiency. The method’s stability, scalability, and computational advantages are demonstrated through empirical validation on income data (exhibiting covariate shift) and housing data (exhibiting prior probability shift), offering theoretical support for standardized reporting of AD in clinical trial publications.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Privacy
• Data Mining & Knowledge Management: Data Visualization & Summarization

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Integrated IPD-AD analysis, which combines individual participant data (IPD) with aggregate data (AD), is increasingly recognized as an effective strategy for generating more reliable and generalizable inferences from heterogeneous studies. While most existing work has focused on algorithmic approaches, this paper investigates a complementary yet underexplored question: how different forms of AD influence the efficiency of data integration. Working within a constrained maximum likelihood estimation framework, we compare commonly reported summary statistics and show that subgroup-specific summaries can substantially improve estimation efficiency. In particular, we find that AD derived from outcome-stratified subgroups (e.g., cases and controls) consistently yield greater efficiency gains than those based on covariate-stratified subgroups (e.g., age or exposure categories), especially when the outcome is continuous. Although outcome-stratified summaries are commonly reported for discrete outcomes, they are rarely provided when the outcome is continuous. Our findings therefore support the routine inclusion of outcome-stratified summaries for continuous endpoints in trial reports and public data repositories to facilitate more efficient evidence synthesis. We further extend the constrained maximum likelihood framework to accommodate dataset shift and develop a fast, non-iterative estimation procedure to improve numerical stability and scalability. We illustrate the proposed methodology with two applications: an analysis of income data under covariate shift and an analysis of housing data under prior probability shift.

Problem

• individual participant data
• aggregate data
• dataset shift
• summary statistics
• data integration

Innovation

• individual participant data
• aggregate data
• outcome-stratified summary
• dataset shift
• constrained maximum likelihood