Quantum Circuit Structure Optimization for Quantum Reinforcement Learning

2025-07-01

AI Summary

In quantum reinforcement learning (QRL), parameterized quantum circuits (PQCs) are typically designed heuristically, lacking theoretical guarantees of optimality and hindering efficient high-dimensional policy learning. Method: We propose QRL-NAS—the first differentiable quantum neural architecture search (QNAST) framework tailored for QRL—that jointly optimizes quantum gate selection and placement to enable end-to-end automated discovery of PQC architectures. Integrating quantum neural networks with reinforcement learning, it leverages the linear and nonlinear transformation capabilities of quantum gates and employs gradient-based architecture search for efficient exploration of the quantum circuit space. Contribution/Results: Experiments across multiple benchmark tasks demonstrate that QRL-NAS significantly improves cumulative reward (average +23.6%) over fixed-architecture baselines, validating its effectiveness and generalizability. This work overcomes the heuristic-design bottleneck and establishes a new paradigm for scalable, high-performance QRL.

Technology Category

• Machine Learning: Quantum Machine Learning
• Search and Optimization: Learning to Search
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Reinforcement learning (RL) enables agents to learn optimal policies through environmental interaction. However, RL suffers from reduced learning efficiency due to the curse of dimensionality in high-dimensional spaces. Quantum reinforcement learning (QRL) addresses this issue by leveraging superposition and entanglement in quantum computing, allowing efficient handling of high-dimensional problems with fewer resources. QRL combines quantum neural networks (QNNs) with RL, where the parameterized quantum circuit (PQC) acts as the core computational module. The PQC performs linear and nonlinear transformations through gate operations, similar to hidden layers in classical neural networks. Previous QRL studies, however, have used fixed PQC structures based on empirical intuition without verifying their optimality. This paper proposes a QRL-NAS algorithm that integrates quantum neural architecture search (QNAS) to optimize PQC structures within QRL. Experiments demonstrate that QRL-NAS achieves higher rewards than QRL with fixed circuits, validating its effectiveness and practical utility.

Problem

• Optimizing quantum circuit structures for quantum reinforcement learning
• Addressing dimensionality curse in RL via quantum computing advantages
• Automating PQC design to improve QRL performance and efficiency

Innovation

• Quantum neural architecture search optimizes circuits
• Parameterized quantum circuits enhance reinforcement learning
• Dynamic circuit structures improve learning efficiency