When Surveys Become Conversations: Adaptive Matrix Validation for AI-Assisted Interviews

2026-06-23

AI Summary

This study addresses measurement error introduced when mapping natural language responses to structured variables in AI-assisted interviews, particularly noting its heterogeneous impact across subpopulations. The authors propose the Adaptive Matrix Validation (AMV) framework, which employs a small random subset of structured validation questions and integrates both cross-respondent calibration and within-respondent verification to doubly correct AI-derived mappings. AMV is the first approach to jointly leverage sparse validation data and inter-respondent calibration, enabling unbiased inference for population means, subgroup parameters, and regression coefficients. The framework also provides a joint planning formula for determining the required number of validation items and sample size. Empirical evaluations—including design-based simulations, a replication using the American Time Use Survey, and an application to CHAMPS verbal autopsy data—demonstrate that AMV substantially improves estimation accuracy even with limited validation resources.

Abstract

AI-assisted interviews promise to reduce respondent burden in surveys by allowing respondents to describe experiences naturally while an AI system noisily maps those accounts into structured survey variables. That mapping is a measurement process that is fallible, versioned, adaptive, and potentially behaves differently across subgroups. This paper proposes Adaptive Matrix Validation (AMV), a design in which each respondent completes an AI-assisted interview, which is then mapped into tabular data by the AI. Respondents are also asked a small, randomized set of structured questions, which are used for statistical adjustment. The estimator first calibrates the mapped values using validation answers from other respondents, then corrects the remaining error with the validation answers observed for the target respondent. The paper develops estimators for item means, subgroup estimates, and regression coefficients when outcomes, predictors, or both are mapped from interviews. It also gives planning formulas the number of validation questions required and the sample size. A design-calibration simulation, an American Time Use Survey emulation, and a CHAMPS verbal-autopsy narrative study show when sparse validation can improve precision and when it cannot

Problem

• AI-assisted interviews
• measurement error
• adaptive validation
• survey methodology
• structured data mapping

Innovation

• Adaptive Matrix Validation
• AI-assisted interviews
• measurement error correction
• survey methodology
• statistical calibration