PennySynth: RAG-Driven Data Synthesis for Automated Quantum Code Generation

2026-05-25

AI Summary

This work addresses the challenges of hallucination, incorrect device configurations, and invalid circuit structures commonly exhibited by general-purpose large language models when generating PennyLane quantum code. To mitigate these issues, the authors propose a retrieval-augmented generation (RAG) framework tailored for quantum programming, leveraging a curated knowledge base of 13,389 verified instruction–code pairs. They introduce a code-aware embedding strategy and adapt the CodeBLEU metric to align with PennyLane’s qml.* API patterns, revealing the complementary roles of structural similarity and functional correctness in evaluation. Using st-codesearch-distilroberta-base for retrieval and a three-stage filtering pipeline, the method achieves pass@5 accuracy rates of 52%–68% on 74 problems from the QHack competitions held between 2022 and 2024, outperforming the non-retrieval Claude Sonnet baseline by 25–28 percentage points.

Abstract

The growing complexity of quantum programming frameworks has exposed a critical limitation in existing large language model (LLM)-based code assistants: general-purpose models hallucinate PennyLane-specific gate names, misplace device configurations, and produce structurally invalid circuits when faced with specialized quantum coding challenges. We present PennySynth, a retrieval-augmented generation framework that addresses this gap by conditioning LLM inference on a curated knowledge base of 13,389 PennyLane instruction-code pairs, built via a three-stage extraction, verification, and deduplication pipeline over official PennyLane repositories, community GitHub sources, and QHack competition archives. PennySynth introduces a code-aware embedding strategy using st-codesearch-distilroberta-base, trained for natural-language-to-code retrieval, increasing average retrieval cosine similarity from 0.45 to 0.726 compared to a general-purpose baseline. Evaluated across 74 challenges spanning three years of the QHack competition (2022, 2023, 2024), PennySynth achieves 64%, 68%, and 52% pass@5 on QHack 2022, 2023, and 2024, respectively, improving over Claude Sonnet 4.6 without retrieval by +28, +25, and +28 percentage points. We further introduce a quantum-adapted CodeBLEU metric that upweights qml.* token patterns and show that structural code similarity and functional correctness capture distinct aspects of quantum code quality. Controlled ablations reveal that code-aware embeddings are the primary driver of retrieval performance, while dataset expansion and source composition provide additional gains when retrieval quality is sufficiently precise.

Problem

• quantum code generation
• PennyLane
• hallucination
• structured invalidity
• specialized programming frameworks

Innovation

• Retrieval-Augmented Generation
• Quantum Code Synthesis
• Code-Aware Embedding
• PennyLane
• Domain-Specific LLM