Benchmarking Large Language Models on Repairing Qiskit Programs using Bugs4Q

2026-07-09

AI Summary

This study addresses the label silent-flip problem in quantum program repair evaluation, where repair correctness depends on the Qiskit version rather than solely on the benchmark itself. By revalidating the Bugs4Q benchmark across six fixed Qiskit versions, this work reveals for the first time its significant version dependency and advocates for mandatory benchmark revalidation prior to repair assessment. The authors generate repair candidates using GPT-4o-mini, GPT-5o-mini, GPT-5.4, and GPT-5.4-mini, evaluating their cross-version effectiveness through executable tests in multiple Qiskit environments. Results show that GPT-5.4 achieves 48.8% pass@10, with most successful repairs occurring on benchmark entries whose original labels had become invalid—highlighting the critical need for revalidation. A version-locked release of Bugs4Q is concurrently made available.

Abstract

In quantum programs, Bugs4Q is a widely used benchmark containing real quantum defects. However, its evaluation assumes that benchmark labels remain valid and that generated fixes execute in the target environment. We evaluate two Bugs4Q versions containing 67 unique real Qiskit defects, adding executable tests where missing, and re-run all entries across six pinned Qiskit releases (0.25.0, 0.45.0, 1.0.0, 1.1.1, 2.0.0, and 2.3.1). We find that quantum benchmarks can suffer from silent label inversion: entries become invalid without errors when reference fixes stop executing or buggy programs no longer reproduce failures. Thus, correctness depends on the (benchmark, version) pair rather than the benchmark alone. We evaluate four LLMs (GPT-4o-mini, GPT-5o-mini, GPT-5.4, and GPT-5.4-mini), generating up to 10 repair candidates per defect and testing them across all versions. GPT-5.4 achieves the highest pass@10 (48.8%), followed by GPT-5.4-mini (47.3%), GPT-5o-mini (30.3%), and GPT-4o-mini (22.6%). All models perform best on Qiskit 0.45.0 and decline after the Qiskit 1.0 transition. Many failures arise from deprecated or incompatible APIs rather than incorrect repairs, and 64\% of successful repairs occur on entries invalid under the target version. We release a re-validated, version-pinned Bugs4Q benchmark and show that benchmark validation must precede repair evaluation.

Problem

• quantum program repair
• benchmark validity
• silent label inversion
• version compatibility
• LLM evaluation

Innovation

• silent label inversion
• version-pinned benchmark
• quantum program repair
• Bugs4Q revalidation
• LLM evaluation