ViM-Q: Scalable Algorithm-Hardware Co-Design for Vision Mamba Model Inference on FPGA

2026-05-03

AI Summary

This work addresses key challenges in deploying Vision Mamba (ViM) models on edge FPGAs, including static quantization failure due to dynamic activation outliers, difficulty in fitting low-bit weight distributions, and architectural mismatch between the state space model (SSM) scanning structure and FPGA streaming computation. To overcome these issues, the authors propose an algorithm-hardware co-design approach featuring dynamic per-token activation quantization, channel smoothing, and 4-bit block-wise additive power-of-two (APoT) weight quantization. They implement a runtime-configurable FPGA accelerator comprising a multiplier-free linear engine using LUTs and a fine-grained pipelined SSM engine. This is the first demonstration of flexible and efficient ViM inference on edge FPGAs for multi-scale, multi-resolution inputs, achieving on average 4.96× speedup and 59.8× better energy efficiency than a quantized RTX 3090 GPU on the ZCU102 platform.

Abstract

Vision Mamba (ViM) models offer a compelling efficiency advantage over Transformers by leveraging the linear complexity of State Space Models (SSMs), yet efficiently deploying them on FPGAs remains challenging. Linear layers struggle with dynamic activation outliers that render static quantization ineffective, while uniform quantization fails to capture the weight distribution at low bit-widths. Furthermore, while associative scan accelerates SSMs on GPUs, its memory access patterns are misaligned with the streaming dataflow required by FPGAs. To address these challenges, we present ViM-Q, a scalable algorithm-hardware co-design for end-to-end ViM inference on the edge. We introduce a hardware-aware quantization scheme combining dynamic per-token activation quantization and per-channel smoothing to mitigate outliers, alongside a custom 4-bit per-block Additive Power-of-Two (APoT) weight quantization. The models are deployed on a runtime-parameterizable FPGA accelerator featuring a linear engine employing a Lookup-Table (LUT) unit to replace multiplications with shift-add operations, and a fine-grained pipelined SSM engine that parallelizes the state dimension while preserving sequential recurrence. Crucially, the hardware supports runtime configuration, adapting to diverse dimensions and input resolutions across the ViM family. Implemented on an AMD ZCU102 FPGA, ViM-Q achieves an average 4.96x speedup and 59.8x energy efficiency gain over a quantized NVIDIA RTX 3090 GPU baseline for low-batch inference on ViM-tiny. This co-design shows a viable path for deploying ViM models on resource-constrained edge devices.

Problem

• Vision Mamba
• FPGA deployment
• quantization
• State Space Models
• edge inference

Innovation

• algorithm-hardware co-design
• hardware-aware quantization
• FPGA acceleration
• State Space Models
• dynamic per-token quantization