Benchmarking Quantum Software Testing with Scalable Quantum Programs

2026-07-02

AI Summary

This work addresses the lack of reproducible and fairly comparable benchmarks in quantum software testing, which has largely relied on small, hard-coded circuits that poorly reflect real-world development practices. To bridge this gap, the authors introduce Qolumbina—the first scalable benchmark suite for quantum software testing—systematically curated from 40 representative programs sourced from open-source repositories and rigorously refactored with standardized interfaces, test cases, and formal specifications. The study further proposes a novel taxonomy of testing characteristics specific to quantum programs and employs program complexity modeling to enable scalability analysis and systematic evaluation. Covering a diverse range of testing attributes, Qolumbina has already facilitated empirical studies on execution overhead and fault detection capability, revealing the critical influence of backend dependencies on the interpretation of testing outcomes.

Abstract

Quantum software testing (QST) checks whether quantum programs behave according to their intended specifications. A key requirement for QST research is a benchmark that supports rigorous empirical evaluation on programs that are testable and better reflect current software development practices. However, existing studies heavily rely on small hard-coded or circuit-level benchmarks, while available quantum programs are scattered across repositories without clear selection criteria, which limits fair comparison and systematic reproducibility. To this end, we present Qolumbina, a benchmark infrastructure for controlled QST experiments on scalable quantum programs. Qolumbina curates 40 programs from open-source repositories, turns them into test-ready subjects through systematic selection, refactoring, specifications, test case examples, unit tests, and standardized interfaces. We also propose QST-oriented criteria to characterize quantum programs along functionality, output behavior, development complexity, and quantum-specific execution complexity. Using these criteria, our empirical study shows that Qolumbina covers diverse testing-relevant properties and supports scalability analysis beyond fixed-size circuit benchmarks. Through controlled experiments with two recent QST approaches, we demonstrate the feasibility of using Qolumbina for execution-cost and fault-detection studies, and highlight backend-dependent effects that can influence QST result interpretation.

Problem

• quantum software testing
• benchmark
• scalable quantum programs
• empirical evaluation
• reproducibility

Innovation

• quantum software testing
• benchmark infrastructure
• scalable quantum programs
• empirical evaluation
• testability