Rethinking Scaffolding in LLM Tutors: The Interactional Mismatch Between Benchmarks and Real-World Deployments

2026-06-14

AI Summary

This study challenges the prevailing assumption in existing AI teaching assistant evaluation benchmarks that students readily adopt scaffolded guidance, an assumption rarely validated in real-world settings. The authors propose a dual-dimensional evaluation framework integrating both “tutor scaffolding” and “student uptake,” analyzing 9,490 authentic educational dialogues across nine datasets. Their quantitative and dialogic analyses reveal that students substantially reduce their reliance on scaffolding in practice and frequently steer conversations autonomously to pursue personalized learning objectives. Crucially, bypassing scaffolding does not necessarily impair learning outcomes; instead, it highlights a misalignment between current instructional frameworks and learners’ goals. These findings question dominant evaluation paradigms and advocate for future benchmarks that prioritize the adaptive capacity of AI teaching assistants across diverse learning contexts.

Abstract

A central pedagogical value evaluated in AI tutor benchmarks is scaffolding: guiding students through graduated steps toward a solution. Alignment and evaluation methods for embedding scaffolding behaviour into chatbots, however, rest on an implicit assumption: that students will take up the scaffolding and engage in the conversation. To examine whether this assumption holds, we introduce an evaluation pipeline around two metrics - Chatbot Scaffolding and Student Uptake - and apply them across nine datasets of 9,490 chats, spanning AI tutor benchmarks and real-world deployments of educational chatbots. Our analysis reveals that while benchmarks assume a high-scaffolding, high-student-uptake environment, students in real-world settings exhibit lower levels of uptake overall - frequently bypassing the chatbot's pedagogical framing to drive the interaction toward their own learning goals at little interpersonal cost. We argue that bypassing scaffolding is not necessarily detrimental; rather, it frequently highlights a mismatch between a chatbot's pedagogical framing and the student's learning goals. To meaningfully evaluate the effectiveness of a chatbot's assistance, future benchmarks must move beyond the assumption that students will simply take up the scaffolding, and instead evaluate how these chatbots navigate diverse learning contexts and student-driven interaction patterns.

Problem

• scaffolding
• student uptake
• AI tutor benchmarks
• real-world deployments
• interactional mismatch

Innovation

• scaffolding
• student uptake
• LLM tutors
• evaluation benchmarks
• real-world deployment