FlexViT: A Flexible FPGA-based Accelerator for Edge Vision Transformers

2026-06-30

AI Summary

Deploying Vision Transformers (ViTs) on edge devices faces significant challenges due to their computational intensity and highly variable tensor shapes, which hinder efficient FPGA resource utilization. This work proposes a hardware-software co-design approach built upon the SECDA-TFLite framework, unifying convolutional and fully connected layers into an INT8 GEMM engine via runtime im2col transformation. A reconfigurable compute array enables dynamic dataflow switching, augmented by a novel dual-mode dataflow mechanism and a depth-first tiling strategy that supports flexible runtime configuration. This design completes accumulation in a single pass, eliminating off-chip partial sum storage and transfers, thereby substantially reducing memory bandwidth requirements. Implemented on the PYNQ-Z2 platform, the proposed accelerator achieves up to 2.74× speedup for individual layers and 1.40× end-to-end inference acceleration over a CPU-only baseline.

Abstract

Deploying Vision Transformer (ViT) models on edge platforms remains challenging due to their high computational demands and the architectural heterogeneity of modern hybrid ViT models, which incorporate both fully connected and convolutional layers. This heterogeneity leads to significant variation in tensor shapes, requiring flexible and efficient FPGA-based acceleration. In this paper, we present FlexViT, a reconfigurable FPGA accelerator for efficient ViT inference on resource-constrained edge devices. Built on the SECDA-TFLite framework, FlexViT employs a hardware-software co-design approach that maps both fully connected and convolutional layers onto a unified high-throughput INT8 GEMM engine using a runtime im2col transformation. To efficiently support diverse layer configurations, we propose a dual-mode dataflow that dynamically switches between input and weight reuse by reconfiguring the compute array at runtime. We further introduce a depth-first tiling strategy that completes accumulation in a single pass, eliminating off-chip partial-sum transfers and reducing memory bandwidth requirements. We implement FlexViT on a PYNQ-Z2 FPGA and evaluate it across a representative set of ViT models. FlexViT achieves up to 2.74x speedup on accelerator-executed layers, translating into up to 1.40x end-to-end speedup compared to CPU-only execution. The code is available at: https://github.com/gicLAB/FlexViT

Problem

• Vision Transformer
• Edge Computing
• FPGA Acceleration
• Model Heterogeneity
• Tensor Shape Variation

Innovation

• FPGA acceleration
• Vision Transformers
• hardware-software co-design
• GEMM engine
• dataflow reconfiguration