Assessing Tenstorrent's RISC-V MatMul Acceleration Capabilities

2025-05-09

AI Summary

This work systematically evaluates the performance and energy efficiency of the Tenstorrent Grayskull e75—a RISC-V-based AI accelerator—on BF16-precision matrix multiplication, targeting its applicability to core linear algebra operations in large language models (LLMs). To this end, we propose the first RISC-V-native MatMul execution model tailored to Grayskull’s hardware architecture, incorporating fine-grained on-die mesh scheduling, BF16-specific optimizations, and empirical benchmarking against established platforms. Experimental results demonstrate a peak energy efficiency of 1.55 TFLOPs/Watt (BF16), outperforming contemporary CPUs (e.g., Intel Sapphire Rapids) under power constraints and matching high-end GPUs (V100/A100). Our key contribution lies in empirically establishing RISC-V’s unique capability to balance architectural flexibility with high energy efficiency in AI acceleration—thereby offering a novel hardware design paradigm for low-precision LLM inference.

Technology Category

• Machine Learning: Hardware-aware ML
• Humans and AI: Other Foundations of Human Computation & AI
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The increasing demand for generative AI as Large Language Models (LLMs) services has driven the need for specialized hardware architectures that optimize computational efficiency and energy consumption. This paper evaluates the performance of the Tenstorrent Grayskull e75 RISC-V accelerator for basic linear algebra kernels at reduced numerical precision, a fundamental operation in LLM computations. We present a detailed characterization of Grayskull's execution model, gridsize, matrix dimensions, data formats, and numerical precision impact computational efficiency. Furthermore, we compare Grayskull's performance against state-of-the-art architectures with tensor acceleration, including Intel Sapphire Rapids processors and two NVIDIA GPUs (V100 and A100). Whilst NVIDIA GPUs dominate raw performance, Grayskull demonstrates a competitive trade-off between power consumption and computational throughput, reaching a peak of 1.55 TFLOPs/Watt with BF16.

Problem

• Evaluating Tenstorrent Grayskull's RISC-V accelerator for AI efficiency
• Assessing matrix operation performance at reduced numerical precision
• Comparing power-performance trade-offs against Intel and NVIDIA hardware

Innovation

• RISC-V accelerator for efficient MatMul operations
• Optimized BF16 precision for energy efficiency
• Competitive performance-power trade-off versus GPUs