Simulating Students' Java Programming Errors with Large Language Models

2026-06-12

AI Summary

This study addresses the high cost of acquiring authentic student error samples in programming education research by systematically evaluating the capacity of large language models (LLMs) to simulate Java logical errors. Leveraging three prompting strategies—Input-Output, Chain-of-Thought, and Self-Refine—the authors assess five LLMs on the CodeWorkout dataset. Results indicate that Claude Sonnet 4 achieves the best performance, and expert blind evaluation confirms that synthetically generated errors are functionally indistinguishable from real student errors. Task difficulty significantly influences simulation fidelity: higher-difficulty tasks yield more diverse errors, albeit with slightly reduced similarity to actual student mistakes. This work demonstrates a viable, low-cost approach to constructing high-quality programming error datasets for educational research.

Abstract

Understanding student errors in the programming is a cornerstone of programming education, yet obtaining a representative set of student errors for any newly designed task remains slow and costly, since authentic submissions only accumulate after extensive classroom deployment. This paper explores whether large language models (LLMs) can serve as scalable proxies for students by simulating realistic logical errors in code submissions. Using the CodeWorkout dataset of 74,000+ unique student Java submissions across 37 problems, we evaluate five LLMs under three mainstream prompting strategies: Input-Output (IO), Chain-of-Thought (CoT), and iterative Self-Refine. We assess performance along two key dimensions: diversity (the range of distinct error patterns) and alignment (alignment with authentic student mistakes), and examine how these vary by struggling level of programming tasks. Our quantitative findings reveal that while all models generate diverse errors, their alignment to human submissions diverges: Claude Sonnet 4 achieves the most balanced performance. In addition, we conducted a blinded expert annotation study (N = 401) comparing synthetic and authentic errors. This qualitative analysis confirms that the generated errors are functionally indistinguishable from authentic student errors. Moreover, higher-struggling-level problems elicit more diverse but less student-like errors. These results highlight trade-offs in using LLMs to simulate human learners and suggest design considerations for integrating synthetic errors into teachable agents, intelligent tutoring systems, and large-scale learning analytics.

Problem

• student errors
• programming education
• code simulation
• large language models
• error diversity

Innovation

• large language models
• student error simulation
• programming education
• error alignment
• synthetic code generation