GraphLeap: Decoupling Graph Construction and Convolution for Vision GNN Acceleration on FPGA

2026-04-23

AI Summary

Vision GNN inference suffers from high computational overhead and serial bottlenecks due to dynamic graph construction at each layer. This work proposes a first-order lookahead graph construction mechanism that decouples graph building from feature updating: the graph for the current layer is constructed using features from the previous layer, while the graph for the next layer is pre-constructed using current-layer features, enabling parallel execution of graph construction and message passing. A streaming FPGA-based architecture is designed, integrating kNN graph construction and feature update engines with node- and channel-level parallelism, eliminating explicit edge storage. Implemented on an Alveo U280, the approach achieves up to 95.7× speedup over CPU and 8.5× over GPU, delivering the first end-to-end pipelined acceleration for Vision GNNs while preserving model accuracy and enabling real-time inference.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Graph-based Machine Learning
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Vision Graph Neural Networks (ViGs) represent an image as a graph of patch tokens, enabling adaptive, feature-driven neighborhoods. Unlike CNNs with fixed grid biases or Vision Transformers with global token interactions, ViGs rely on dynamic graph convolution: at each layer, a feature-dependent graph is built via k-nearest-neighbor (kNN) search on current patch features, followed by message passing. This per-layer graph construction is the main bottleneck, consuming 50--95\% of graph convolution time on CPUs and GPUs, scaling as $O(N^2)$ with the number of patches $N$, and creating a sequential dependency between graph construction and feature updates. We introduce GraphLeap, a simple reformulation that removes this dependency by decoupling graph construction from feature update across layers. GraphLeap performs the feature update at layer $\ell$ using a graph built from the previous layer's features, while simultaneously using the current layer's features to construct the graph for layer $\ell+1$. This one-layer-lookahead graph construction enables concurrent graph construction and message passing. Although using prior-layer features can introduce minor accuracy degradation, lightweight fine-tuning for a few epochs is sufficient to recover the original accuracy. Building on GraphLeap, we present the first end-to-end FPGA accelerator for Vision GNNs. Our streaming, layer-pipelined design overlaps a kNN graph construction engine with a feature update engine, exploits node- and channel-level parallelism, and enables efficient on-chip dataflow without explicit edge-feature materialization. Evaluated on isotropic and pyramidal ViG models on an Alveo U280 FPGA, GraphLeap achieves up to $95.7\times$ speedup over CPU and $8.5\times$ speedup over GPU baselines, demonstrating the feasibility of real-time Vision GNN inference.

Problem

• Vision Graph Neural Networks
• graph construction
• k-nearest-neighbor search
• computational bottleneck
• sequential dependency

Innovation

• GraphLeap
• Vision GNN
• graph decoupling
• FPGA acceleration
• kNN graph construction