Evaluating Agentic Code Repair Capabilities in Distributed Systems

2026-08-14

AI Summary

This study addresses the absence of code repair benchmarks and the unclear impact of debugging context in distributed systems by introducing DDBench, the first dedicated benchmark for this domain. Constructed through historical bug mining and hierarchical generation, DDBench employs dual-condition controlled experiments with symptom and context augmentation to decouple model capability from context utility. Results reveal a 61 percentage point variance in LLM repair pass rates and an asymmetric gain of 18.1 percentage points attributable to debugging context. By filling a critical evaluation gap, this work elucidates the pivotal role of context in agent-based repair, providing empirical evidence and methodological foundations for intelligent debugging in distributed systems.

Abstract

LLM-based coding agents have advanced rapidly on single-process SWE tasks, with frontier models now clustering in the high-70s on SWE-bench Verified. Distributed-system debugging, however, remains an under-explored regime: bugs span processes, nodes, and protocol interactions, with root causes rarely recoverable from source alone and brute-force exploration intractable across non-deterministic interleavings. This leaves two gaps in LLM and agent evaluation: no code-repair benchmark targets distributed-system bugs, and no controlled study isolates how much externally provided debugging context changes agent success on them. We introduce DDBench, a code-repair benchmark of 60 historical bugs mined from 13 open-source distributed systems, partitioned into three difficulty tiers. DDBench evaluates every case under two matched conditions: a symptom-only condition where the agent receives only the bug symptom and repository, and a context-augmented condition where it additionally receives a bounded debugging context (logs, traces, runtime state, and targeted code-investigation notes), isolating the effect of debugging context from model capability. The evaluation of ten LLMs on DDBench reveals several findings. First, distributed debugging exercises a reasoning dimension that single-process benchmarks do not surface: models' pass rates span 61 pp, and pairwise bootstrap separates 9 of 15 top-tier model pairs at p < 0.05 on DDBench's hardest case-set. Second, bounded debugging context lifts aggregate pass rate by +18.1 pp, and the lift is asymmetric: weaker models gain pass rate, while stronger models gain efficiency. Third, debugging context requires careful curation, as even faithful debugging context can sometimes mislead LLMs.

Problem

• Distributed Systems
• Code Repair
• LLM Agents
• Benchmark
• Debugging Context

Innovation

• DDBench
• Distributed System Debugging
• Code Repair Benchmark
• Context-Augmented Evaluation
• LLM Agents