When the Target Domain Changes: AI-Mediated Construct Drift in High-Stakes English Language AssessmenW

2026-07-13

AI Summary

This study addresses the construct-irrelevance threat emerging in high-stakes English language assessments due to generative AI’s integration into target-language use domains, which creates a misalignment between traditional unassisted scoring and authentic academic communication competence. Introducing the novel concept of “AI-mediated construct drift,” the work frames AI not merely as a tool but as a validity interpretation issue. It advocates a “limited AI mediation” principle—providing all test takers with a standardized, controlled AI assistant under uniform testing conditions, explicitly distinguishing comprehension support from answer generation. Drawing on construct validity theory, AI interaction log analysis, and principles of standardized test design, the research elucidates the mechanisms driving the evolution of language ability constructs in AI-augmented contexts, offering a validity-centered paradigm and actionable design framework for high-stakes language assessment in the AI era.

Abstract

High-stakes English proficiency tests treat standardized, unaided performance as evidence for score interpretations about academic English proficiency. This interpretation remains meaningful, but as target language use domains increasingly involve generative AI, the extrapolation from unaided test performance to academic communicative readiness becomes less self-evident. This conceptual validity argument reframes AI as a score-interpretation problem in high-stakes language testing, not only an operational issue of scoring, feedback, security, or misconduct. Synthesizing current literature in three uneven layers, the paper shows that most work treats AI as assessment infrastructure, while far less theorizes its implications for construct validity and extrapolation warrants. It defines AI-mediated construct drift as the misalignment that arises when communicative abilities required in the target domain change through AI mediation while test constructs remain anchored to an unaided-performance model. It proposes bounded AI mediation as a validity-oriented design principle: a standardized condition in which all test takers access the same institutionally controlled AI assistant, with predefined assistance boundaries, logged interactions, and tasks that distinguish comprehension support from answer generation. The paper argues that score interpretations should be narrowed and supplemented when used to support claims about AI-mediated academic communication.

Problem

• construct drift
• AI-mediated communication
• high-stakes language assessment
• validity argument
• score interpretation

Innovation

• AI-mediated construct drift
• bounded AI mediation
• construct validity
• score interpretation
• generative AI