AP-DRL: A Synergistic Algorithm-Hardware Framework for Automatic Task Partitioning of Deep Reinforcement Learning on Versal ACAP

2026-03-31

AI Summary

This work addresses key challenges in deep reinforcement learning (DRL) training, including inefficient algorithm-hardware co-design, significant disparities in computational intensity across operations, and reward inaccuracies introduced by mixed-precision quantization. To tackle these issues, the authors propose AP-DRL, a framework that enables automated task partitioning and coordinated multi-precision (FP32/FP16/BF16) execution on AMD Versal ACAP heterogeneous platforms. By combining performance bottleneck analysis, design space exploration, and integer linear programming-based modeling, AP-DRL intelligently maps computational operations to the most suitable processing units—CPU, FPGA fabric, or AI Engines—balancing computational efficiency with training convergence. Experimental results demonstrate that AP-DRL achieves up to 4.17× and 3.82× speedup over baseline implementations on programmable logic and AI Engines, respectively, while preserving convergence performance.

Abstract

Deep reinforcement learning has demonstrated remarkable success across various domains. However, the tight coupling between training and inference processes makes accelerating DRL training an essential challenge for DRL optimization. Two key issues hinder efficient DRL training: (1) the significant variation in computational intensity across different DRL algorithms and even among operations within the same algorithm complicates hardware platform selection, while (2) DRL's wide dynamic range could lead to substantial reward errors with conventional FP16+FP32 mixed-precision quantization. While existing work has primarily focused on accelerating DRL for specific computing units or optimizing inference-stage quantization, we propose AP-DRL to address the above challenges. AP-DRL is an automatic task partitioning framework that harnesses the heterogeneous architecture of AMD Versal ACAP (integrating CPUs, FPGAs, and AI Engines) to accelerate DRL training through intelligent hardware-aware optimization. Our approach begins with bottleneck analysis of CPU, FPGA, and AIE performance across diverse DRL workloads, informing the design principles for AP-DRL's inter-component task partitioning and quantization optimization. The framework then addresses the challenge of platform selection through design space exploration-based profiling and ILP-based partitioning models that match operations to optimal computing units based on their computational characteristics. For the quantization challenge, AP-DRL employs a hardware-aware algorithm coordinating FP32 (CPU), FP16 (FPGA/DSP), and BF16 (AI Engine) operations by leveraging Versal ACAP's native support for these precision formats. Comprehensive experiments indicate that AP-DRL can achieve speedup of up to 4.17$\times$ over programmable logic and up to 3.82$\times$ over AI Engine baselines while maintaining training convergence.

Problem

• Deep Reinforcement Learning
• Hardware Acceleration
• Mixed-Precision Quantization
• Task Partitioning
• Computational Intensity

Innovation

• Automatic Task Partitioning
• Hardware-Aware Quantization
• Heterogeneous Acceleration
• Deep Reinforcement Learning
• Versal ACAP