Problem
Research questions and friction points this paper is trying to address.
LLM tutoring
programming education
student help-seeking
response style
productive continuation
When LLM Tutoring Responses Work: Evidence from Student Programming Conversations
🤖 AI Summary
This study investigates which response styles of large language models (LLMs) most effectively promote students’ sustained engagement in programming tutoring. Analyzing 16,851 real student–ChatGPT dialogues, the authors employed the Gemma-4 local model for assisted annotation complemented by human validation to systematically label help-seeking contexts, student states, response styles, and subsequent behaviors. Using chi-square tests and effect size analyses, they quantified the impact of response styles on learning continuation. Results reveal that confirmatory feedback yields the highest continuation rate (82.4%), whereas providing direct answers performs worst (62.7%). Crucially, the optimal response style is highly contingent on students’ cognitive load and level of confusion. This work provides the first empirical evidence from authentic tutoring interactions demonstrating the context-dependent relationship between LLM response styles and learning persistence, offering actionable insights for the design of educational LLM applications.
📝 Abstract
As students increasingly use LLM tutors in computer science education, one question becomes especially important: what kind of response helps a student continue productively? Prior work has studied how students use LLMs in computer science education, but less is known about how tutoring response styles are associated with student follow-up across programming help-seeking contexts. This paper analyzes StudyChat (UMass, 2026), a public dataset of student and ChatGPT tutoring conversations from an artificial intelligence course. We transformed StudyChat into 16,851 assistant-response interactions from 203 students and 2,214 conversations. Using local LLM-assisted annotation with Gemma 4, we labeled student help-seeking situations, student state, assistant response style, and student next-turn outcome. Human validation showed 82\% agreement with the LLM-assisted labels (Cohen's $κ=.74$). We analyzed productive continuation and unresolved continuation across the full dataset and across help-seeking contexts. Globally, response style was significantly associated with productive continuation, $χ^2(7)=100.39$, $p<.001$, $V=.078$, and unresolved continuation, $χ^2(7)=125.77$, $p<.001$, $V=.087$, though effect sizes were small. Verification feedback had the highest productive-continuation rate (82.4\%), while direct answers had the lowest (62.7\%). Descriptively, response-style score ranges were smallest in low-confusion conceptual contexts (.017) and largest in high-cognitive-load contexts (.203). More detailed comparisons showed situation-dependent response patterns. For example, stepwise guidance was followed by greater confusion decrease in high-cognitive-load code requests, while direct answers were followed by more unresolved continuation in high-load debugging. These findings support context-aware evaluation and design of AI tutoring responses for programming education.
Innovation
Methods, ideas, or system contributions that make the work stand out.
context-aware tutoring
LLM-assisted annotation
response style
productive continuation
programming education
🔎 Similar Papers
No similar papers found.
