QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding

2026-04-28

AI Summary

This work addresses the lack of systematic evaluation of existing vision-language models in understanding quantum calibration diagrams. We present the first multimodal benchmark specifically designed for quantum calibration tasks, encompassing diverse scenarios and question types across both superconducting and neutral-atom quantum computing platforms, thereby establishing a standardized evaluation framework. Through comprehensive assessments using zero-shot inference, in-context learning, and supervised fine-tuning, we systematically evaluate state-of-the-art models and uncover significant performance gaps in multi-image contextual reasoning. The best-performing general-purpose zero-shot model achieves an average score of 72.3, while our newly released open-source model, NVIDIA Ising Calibration 1, attains 74.7 under zero-shot settings.

Abstract

Quantum computing calibration depends on interpreting experimental data, and calibration plots provide the most universal human-readable representation for this task, yet no systematic evaluation exists of how well vision-language models (VLMs) interpret them. We introduce QCalEval, the first VLM benchmark for quantum calibration plots: 243 samples across 87 scenario types from 22 experiment families, spanning superconducting qubits and neutral atoms, evaluated on six question types in both zero-shot and in-context learning settings. The best general-purpose zero-shot model reaches a mean score of 72.3, and many open-weight models degrade under multi-image in-context learning, whereas frontier closed models improve substantially. A supervised fine-tuning ablation at the 9-billion-parameter scale shows that SFT improves zero-shot performance but cannot close the multimodal in-context learning gap. As a reference case study, we release NVIDIA Ising Calibration 1, an open-weight model based on Qwen3.5-35B-A3B that reaches 74.7 zero-shot average score.

Problem

• quantum calibration
• calibration plots
• vision-language models
• benchmarking
• multimodal understanding

Innovation

• vision-language models
• quantum calibration
• benchmark
• in-context learning
• multimodal evaluation