Problem
Research questions and friction points this paper is trying to address.
LLM-based measures
student discourse
epistemic authority
marginalized youth
re-contextualization
When Youth Enter The Chat: An Epistemic Shift in the Validation of LLM-Based Measures of Student Talk
🤖 AI Summary
研究通过多方法重新背景化学生对话，让青少年参与研究过程，解决基于LLM的学生话语测量中存在的去背景化和验证不足问题。
📝 Abstract
LLMs are being used increasingly to measure aspects of student discourse (e.g. talk moves, collaboration, equity of voice) at scale. Typically, LLM-based measures of student talk use transcriptions of classroom conversations that only include verbal contributions, which de-contextualize student language. Common practices for validating these measures include comparing outputs against expert annotations by adults, using held out evaluation sets and F1 scores. We argue that these approaches are insufficient to ensure that such measures are meaningful and equitable for teaching and learning, particularly for racially and linguistically marginalized youth. In order to center the youth whose talk is being analyzed, re-contextualizing these classroom conversations and engaging youth in the research process is necessary. Sharing epistemic authority with youth, ultimately, centers their point of view and adds crucial nuance to the analysis of their talk that adult experts, researchers, and LLMs cannot provide. In a case study of multilingual youth in one 8th-grade math classroom, we address the epistemic exclusion of youth by employing multiple ethnographically-oriented methods to re-contextualize student conversations and center youth as epistemic authorities in conversation with researchers and LLMs. We conducted participant observations, interviews, focus groups, and member checks with four focal students. Findings reveal that there were misalignments between students' interpretations of their own math talk experiences and the LLM-based measures of their talk. Students contested both the LLM classifications and the coding scheme used to measure their talk, highlighting the need for youth to be involved in the epistemic process of producing knowledge about their experiences.
Innovation
Methods, ideas, or system contributions that make the work stand out.
youth participation
re-contextualization
epistemic authority
🔎 Similar Papers
No similar papers found.
