Your Students Don't Use LLMs Like You Wish They Did

2026-04-25

AI Summary

This study addresses the lack of effective methods for evaluating the alignment between student–AI dialogues and instructional objectives in educational NLP systems. Proposing six turn-level computational metrics, the work leverages dialogue analysis and NLP techniques to model student behavioral patterns at a fine-grained level, based on 12,650 utterances across 500 lesson dialogues. The research reveals, for the first time, that students predominantly use AI for answer extraction rather than sustained learning, and identifies deployment context as a key factor shaping usage patterns. The proposed metrics effectively detect misalignments between pedagogical design and actual student behavior, offering a quantifiable and actionable tool for assessing the extent to which instructional goals are achieved in educational conversational systems.

Technology Category

• Natural Language Processing: Conversational AI/Dialog Systems
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Educational NLP systems are typically evaluated using engagement metrics and satisfaction surveys, which are at best a proxy for meeting pedagogical goals. We introduce six computational metrics for automated evaluation of pedagogical alignment in student-AI dialogue. We validate our metrics through analysis of 12,650 messages across 500 conversations from four courses. Using our metrics, we identify a fundamental misalignment: educators design conversational tutors for sustained learning dialogue, but students mainly use them for answer-extraction. Deployment context is the strongest predictor of usage patterns, outweighing student preference or system design: when AI tools are optional, usage concentrates around deadlines; when integrated into course structure, students ask for solutions to verbatim assignment questions. Whole-dialogue evaluation misses these turn-by-turn patterns. Our metrics will enable researchers building educational dialogue systems to measure whether they are achieving their pedagogical goals.

Problem

• pedagogical alignment
• student-AI dialogue
• educational NLP
• answer-extraction
• dialogue systems

Innovation

• pedagogical alignment
• educational dialogue systems
• computational metrics
• student-AI interaction
• automated evaluation