Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?

2026-04-19

AI Summary

This work addresses the challenge that large language models often over-edit code during debugging, leading to imprecise fixes and a lack of effective metrics for evaluating debugging accuracy. To this end, the authors propose the Precise Debugging Benchmark (PDB) framework, which automatically injects atomic-level errors into arbitrary programming datasets and composes them into single- and multi-error programs to create precision-aware debugging benchmarks. PDB introduces two novel evaluation metrics—edit-level precision and bug-level recall—and constructs challenging evaluation sets, PDB-Single-Hard and PDB-Multi. Experimental results show that state-of-the-art models such as GPT-5.1-Codex and DeepSeek-V3.2-Thinking achieve unit test pass rates exceeding 76%, yet their edit-level precision remains below 45%. Moreover, iterative refinement and agent-based strategies do not yield significant performance improvements.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during debugging. To evaluate how far LLMs are from precise debugging, we introduce the Precise Debugging Benchmark (PDB) framework, which automatically converts any coding dataset into a debugging benchmark with precision-aware evaluation. PDB generates buggy programs by synthesizing verified atomic bugs and composing them into multi-bug programs. We define two novel metrics, edit-level precision and bug-level recall, which measures how many necessary edits are made and how many bugs are resolved. We release two evaluation benchmarks: PDB-Single-Hard on single-line bugs, and PDB-Multi on multi-line bugs. Experiments show that frontier models, such as GPT-5.1-Codex and DeepSeek-V3.2-Thinking, achieve unit-test pass rates above 76% but exhibit precision below 45%, even when explicitly instructed to perform minimal debugging. Finally, we show that iterative and agentic debugging strategies do not substantially improve precision or recall, highlighting the need to rethink post-training pipelines for coding models.

Problem

• precise debugging
• code debugging
• over-editing
• fault localization
• minimal repair

Innovation

• Precise Debugging Benchmark
• edit-level precision
• bug-level recall
• atomic bugs
• LLM debugging evaluation