Measuring Whether LLM Tutors Teach or Solve: A Diagnostic for Educational Impact

2026-06-15

AI Summary

This study addresses a critical limitation in current educational evaluations of large language models (LLMs), which often conflate problem-solving proficiency with pedagogical support capability by equating correct answer generation with effective teaching. To disentangle these dimensions, the authors propose a lightweight diagnostic framework built upon the publicly available MathTutorBench benchmark, establishing a dual-axis scoring system that separately assesses solution accuracy and instructional quality. Their analysis reveals only a weak correlation between the two dimensions (r = 0.421), demonstrating for the first time that strong performance in solving problems does not necessarily translate to effective teaching. Evaluations across eight prominent LLMs show substantial ranking shifts under the two assessment criteria, underscoring the necessity of distinct evaluation protocols. The work advocates centering student agency—through indicators such as guided questioning and calibrated scaffolding—as the core metric, urging a paradigm shift in educational AI evaluation from “getting the right answer” to “enabling the learner to understand.”

Abstract

Large language models are increasingly proposed as educational tutors, yet stronger task-solving ability does not necessarily imply stronger learning support. Motivated by recent calls to measure the social impact of NLP systems in practice, we study whether public LLM tutoring benchmarks distinguish learning-supportive behavior from mere answer production. We propose a lightweight diagnostic based on the gap between solving-oriented and pedagogy-oriented benchmark performance. Using public MathTutorBench leaderboard results, we show that these dimensions are only partially aligned: across eight publicly reported models, the correlation between solving and pedagogy composites is 0.421, and several models shift meaningfully in rank when evaluation moves from solving to pedagogy. We then analyze the public TutorBench sample and show that agency-relevant behaviors are explicitly encoded in benchmark rubrics, especially in active-learning settings that reward guiding questions, calibrated hints, and non-disclosive scaffolding. Together, these findings suggest that educational-impact evaluation should not treat task success as a sufficient proxy for learning support. We argue that public tutoring benchmarks can better support positive-impact evaluation by reporting solving-oriented and pedagogy-oriented scores separately and by making disclosure-sensitive, student-agency-preserving criteria more explicit.

Problem

• educational impact
• large language models
• tutoring benchmarks
• learning support
• student agency

Innovation

• educational impact
• pedagogy-oriented evaluation
• student agency
• scaffolding
• LLM tutoring