PyMETA: A Benchmark Dataset for Hierarchical Student Code Error Classification with Python-Interpreter-Based Labels

2026-06-29

AI Summary

This study addresses the limitations of existing code error detection methods—namely, the absence of large-scale datasets, insufficient capability for multi-error analysis, and lack of a unified classification framework—which hinder context-sensitive debugging in educational settings. To bridge this gap, the authors propose a three-tier hierarchical error taxonomy grounded in Python’s official exception hierarchy and introduce PyMETA, the first large-scale, fine-grained annotated dataset of student code errors, comprising 48,646 submissions and 97 expert-annotated multi-error samples. Leveraging static analysis, expert annotation, and prompt engineering, the work systematically evaluates fine-tuned models (e.g., CodeBERT) against prominent large language models (e.g., GPT-3.5, Gemini 2.5 Pro) across multiple error recognition tasks. Experiments show that Gemini 2.5 Pro achieves a macro F1 score of 81.8% under a multi-error “inclusion” criterion, yet prompt-based LLMs generally underperform fine-tuned smaller models and exhibit a consistent tendency to over-predict logical errors.

Abstract

With the advancement of Large Language Models (LLMs), code error detection has extended beyond traditional IDE diagnostics to context-sensitive debugging in educational scenarios. However, existing approaches lack large-scale datasets, multi-error analysis, and unified error taxonomies. To address this, we introduce PyMETA, a large-scale Python code error classification dataset of 48,646 student submissions, with single-error labels for all samples and a diagnostic subset of 97 expert-annotated multi-error samples. The dataset uses a three-level hierarchical taxonomy, from a binary error/no-error split down to 14 fine-grained error types grounded in Python's official exception hierarchy. We evaluate multi-level classification tasks on two finetuned models and four LLMs with prompting, comparing their classification performance and runtime cost. For multi-error prompting, the best model, Gemini 2.5 Pro, achieves 81.8% macro F1 under the "contains" criterion. We observe that: 1) prompted LLMs still underperform finetuned smaller models; 2) models exhibit significant disparities across error types; 3) most LLMs over-classify code as Logic Error, with GPT-3.5 showing the highest Logic Error Overprediction Rate and Gemini 2.5 Pro the lowest. Our work establishes a data foundation and provides insights for LLM-based code error research.

Problem

• code error classification
• student code
• hierarchical taxonomy
• large language models
• educational debugging

Innovation

• hierarchical error taxonomy
• Python interpreter-based labeling
• multi-error classification
• student code dataset
• LLM evaluation for debugging