ELiTeFormer: An Efficient Transformer for FPGAs

2026-07-03

AI Summary

This work addresses the high computational and memory demands of deploying Transformer-based large language models on FPGAs. The authors propose ELiTeFormer, an FPGA-aware efficient Transformer architecture that uniquely integrates linear attention with ternary linear projections, eliminating multiplications entirely and thereby avoiding DSP resource utilization. Through ternary quantization, bitmasked operations, and a high-level synthesis (HLS) design flow, the approach achieves algorithm-hardware co-optimization on the Xilinx VCK5000 platform. Compared to LLaMA 3, ELiTeFormer compresses model weights by 10× and KV cache by 12.8× while achieving a 31.9% accuracy on the MMLU benchmark. It further reduces end-to-end inference latency by 3.9× and improves energy efficiency by 3.2×.

Abstract

Transformer blocks are prevalent in large language model (LLM) but present deployment challenges due to their challenging computational and memory demands. While prior work has typically optimized attention mechanisms or feed-forward networks (FFNs) separately, few hardware (HW) architecture have jointly addressed both components with co-designed hardware acceleration. We present ELiTeFormer (Efficient Linear Ternary Transformer), the first Transformer model architecture that unifies hybrid linear attention with ultra-low-precision (ternary) linear projections, specifically co-designed for field-programmable gate array (FPGA) deployment. ELiTeFormer achieves 10x model weight compression and 12.8x key-value (KV) cache compression compared to LLaMA 3, while maintaining competitive accuracy (31.9% on the MMLU benchmark, within 3.0% of BitNet b1.58). Our key architectural contribution is a novel processing element (PE) micro-architecture that eliminates all multiplications in ternary linear projections through bitmasking operations, significantly reducing resource utilization by completely avoiding dedicated digital signal processing (DSP) blocks. We simulate, synthesize, and deploy ELiTeFormer targeting a Xilinx VCK5000 Versal board using high-level synthesis (HLS) flows. Block-level simulations show 9.6x speedup for FFN operations and 4.4x speedup for attention compared to standard implementations. End-to-end deployment achieves up to 3.9x lower latency and 3.2x better energy efficiency than LLaMA 3 on an NVIDIA A100 graphics processing unit (GPU) at long context lengths. This represents the first FPGA realization combining linear attention with ternary quantization, demonstrating the viability of algorithm-architecture co-design for next-generation LLM acceleration.

Problem

• Transformer
• FPGA deployment
• hardware acceleration
• model compression
• large language models

Innovation

• Ternary Quantization
• Linear Attention
• FPGA Acceleration
• Algorithm-Architecture Co-design
• Multiplication-Free PE