Real Time FPGA Based CNNs for Detection, Classification, and Tracking in Autonomous Systems: State of the Art Designs and Optimizations

2025-09-04

AI Summary

Deploying CNN-based object detection, classification, and tracking on FPGAs for edge applications—such as autonomous driving and robotics—is hindered by stringent hardware resource constraints, low energy efficiency, and high inference latency. Method: This work proposes a hardware-software co-optimization framework integrating model lightweighting (pruning, quantization, sparsification) with heterogeneous architecture design (LUT-DSP collaboration, SoC FPGA/ACAP platforms), enabling dynamic reconfiguration and low-power real-time inference; it further explores GPU-FPGA hybrid acceleration. End-to-end deployment is realized using the Vitis AI and FINN toolchains. Contribution/Results: The framework achieves up to 3.2× higher throughput, sub-15 ms latency, and 68% lower power consumption versus GPU-based solutions—while preserving accuracy. It delivers a scalable, practical methodology for developing high-throughput, ultra-low-power embedded vision systems.

Technology Category

• Machine Learning: Hardware-aware ML
• Computer Vision: Learning & Optimization for CV
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Security and privacy of machine learning and AI applications
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

This paper presents a comprehensive review of recent advances in deploying convolutional neural networks (CNNs) for object detection, classification, and tracking on Field Programmable Gate Arrays (FPGAs). With the increasing demand for real-time computer vision applications in domains such as autonomous vehicles, robotics, and surveillance, FPGAs have emerged as a powerful alternative to GPUs and ASICs due to their reconfigurability, low power consumption, and deterministic latency. We critically examine state-of-the-art FPGA implementations of CNN-based vision tasks, covering algorithmic innovations, hardware acceleration techniques, and the integration of optimization strategies like pruning, quantization, and sparsity-aware methods to maximize performance within hardware constraints. This survey also explores the landscape of modern FPGA platforms, including classical LUT-DSP based architectures, System-on-Chip (SoC) FPGAs, and Adaptive Compute Acceleration Platforms (ACAPs), comparing their capabilities in handling deep learning workloads. Furthermore, we review available software development tools such as Vitis AI, FINN, and Intel FPGA AI Suite, which significantly streamline the design and deployment of AI models on FPGAs. The paper uniquely discusses hybrid architecture that combine GPUs and FPGAs for collaborative acceleration of AI inference, addressing challenges related to energy efficiency and throughput. Additionally, we highlight hardware-software co-design practices, dataflow optimizations, and pipelined processing techniques essential for real-time inference on resource-constrained devices. Through this survey, researchers and engineers are equipped with insights to develop next-generation, power-efficient, and high-performance vision systems optimized for FPGA deployment in edge and embedded applications.

Problem

• Real-time object detection, classification, and tracking using CNNs
• FPGA-based hardware acceleration for autonomous systems applications
• Optimizing CNN performance through pruning, quantization, and co-design techniques

Innovation

• FPGA-based CNN deployment for real-time vision tasks
• Hardware acceleration with pruning, quantization, sparsity optimization
• Hybrid GPU-FPGA architecture for collaborative AI inference