Benchmarking Quantum Computers via Protocols, Comparing IBM's Heron vs IBM's Eagle

2026-03-04

AI Summary

This work addresses the challenge of objectively evaluating whether next-generation quantum processors have achieved practical quantum advantage. The authors propose a protocol-level benchmarking framework that assesses hardware performance not at the gate level but through well-defined “quantumness thresholds” at the protocol level, yielding evaluation criteria that are more application-relevant, transparent, and intuitive. Applying this methodology to empirical comparisons between IBM’s Eagle and Heron processors demonstrates that the Heron architecture delivers significant performance gains on real-world tasks, thereby substantiating its tangible progress toward practical quantum advantage.

Technology Category

• Machine Learning: Quantum Machine Learning
• Search and Optimization: Evaluation and Analysis
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

As quantum computing hardware rapidly advances, objectively evaluating the capabilities and error rates of new processors remains a critical challenge for the field. A clear and realistic understanding of current quantum performance is essential to guide research priorities and drive meaningful progress. In this work, we apply and extend a protocol-based benchmarking methodology (presented in arXiv:2505.12441) that utilizes well-defined quantumness thresholds. By evaluating performance at protocol level rather then the gate level, this approach provides a transparent and intuitive assessment of whether specific quantum processors, or isolated sub-chips within them, can demonstrate a practical quantum advantage. To illustrate the utility of this method, we compare two generations of IBM quantum computers: the older Eagle architecture and the newer Heron architecture. Our findings reveal the genuine operational strengths and limitations of these devices, demonstrating substantial performance improvements in the newer Heron generation.

Problem

• quantum benchmarking
• quantum advantage
• quantum processors
• error rates
• performance evaluation

Innovation

• protocol-based benchmarking
• quantum advantage
• quantumness threshold
• IBM Heron
• IBM Eagle