Handling Missing Responses under Cluster Dependence with Applications to Language Model Evaluation

2025-10-23

AI Summary

This paper addresses two pervasive challenges in human evaluation of generative AI: nonresponse (i.e., missing responses) and clustering dependence (e.g., multiple turns from the same user inducing response correlation). We propose the first doubly robust estimation framework specifically designed for clustered data structures. Theoretically, we establish, for the first time, the unbiasedness, consistency, and asymptotic normality of this estimator under both cluster-level dependence and non-ignorable missingness. Methodologically, we jointly model the response generation mechanism and within-cluster correlation structure to achieve unbiased estimation of average scores and calibrated uncertainty quantification. Experiments on synthetic and real-world dialogue quality datasets demonstrate that ignoring clustering leads to underestimated standard errors and invalid statistical inference, whereas our method substantially improves estimation accuracy and confidence interval coverage. The framework thus provides a statistically rigorous and practically viable foundation for AI evaluation.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Generation
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Human annotations play a crucial role in evaluating the performance of GenAI models. Two common challenges in practice, however, are missing annotations (the response variable of interest) and cluster dependence among human-AI interactions (e.g., questions asked by the same user may be highly correlated). Reliable inference must address both these issues to achieve unbiased estimation and appropriately quantify uncertainty when estimating average scores from human annotations. In this paper, we analyze the doubly robust estimator, a widely used method in missing data analysis and causal inference, applied to this setting and establish novel theoretical properties under cluster dependence. We further illustrate our findings through simulations and a real-world conversation quality dataset. Our theoretical and empirical results underscore the importance of incorporating cluster dependence in missing response problems to perform valid statistical inference.

Problem

• Addressing missing annotations in human evaluation data
• Handling cluster dependence in human-AI interaction correlations
• Achieving unbiased estimation with proper uncertainty quantification

Innovation

• Doubly robust estimator addresses missing annotations
• Novel theoretical properties under cluster dependence
• Incorporating cluster dependence for valid inference