EONSim: An NPU Simulator for On-Chip Memory and Embedding Vector Operations

2025-11-10

AI Summary

Existing NPU simulators support only deterministic matrix computations under simplistic double-buffered memory, failing to model the input-dependent and nondeterministic memory access patterns inherent in embedding operations. To address this gap, we propose the first unified NPU simulation platform supporting both dense matrix computations and sparse embedding vector operations. Our platform introduces fine-grained modeling of complex on-chip memory architectures—including multi-bank memory and heterogeneous caches—as well as diverse memory management strategies such as prefetching, tiling, and dynamic address mapping. Calibrated against TPUv6e hardware, it achieves mean absolute errors of just 1.4% in inference latency and 2.2% in on-chip memory accesses. This work fills a critical void in embedding-intensive workload simulation and significantly enhances the accuracy and practicality of design-space exploration for emerging NPUs.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

Embedding vector operations are a key component of modern deep neural network workloads. Unlike matrix operations with deterministic access patterns, embedding vector operations exhibit input data-dependent and non-deterministic memory accesses. Existing neural processing unit (NPU) simulators focus on matrix computations with simple double-buffered on-chip memory systems, lacking the modeling capability for realistic embedding behavior. Next-generation NPUs, however, call for more flexible on-chip memory architectures that can support diverse access and management schemes required by embedding workloads. To enable flexible exploration and design of emerging NPU architectures, we present EONSim, an NPU simulator that holistically models both matrix and embedding vector operations. EONSim integrates a validated performance model for matrix computations with detailed memory simulation for embedding accesses, supporting various on-chip memory management policies. Validated against TPUv6e, EONSim achieves an average inference time error of 1.4% and an average on-chip memory access count error of 2.2%.

Problem

• Modeling non-deterministic memory accesses in embedding operations
• Supporting flexible on-chip memory architectures for NPUs
• Enabling holistic simulation of matrix and embedding computations

Innovation

• Simulates both matrix and embedding vector operations
• Models flexible on-chip memory management policies
• Integrates validated performance with detailed memory simulation