tensor processing unit

This work addresses critical limitations of existing cycle-accurate simulators for TPU-like systolic arrays—namely, insufficient accuracy, lack of validation against real hardware, and poor integration with modern ML compiler stacks. To overcome these challenges, we extend SCALE-Sim v3 to build a high-fidelity simulation platform tailored for TPUs. Our approach features the first hardware-validated systolic GEMM model on actual TPU v4 silicon, introduces a lightweight tensor-shape-aware latency model for non-systolic operations, and integrates a StableHLO frontend to enable end-to-end simulation from mainstream frameworks such as JAX and PyTorch. Experimental results demonstrate a strong linear correlation between simulated cycles and real TPU v4 execution latency, with median relative error below 3% for non-systolic operations, substantially improving both simulation accuracy and practical utility.

Current TPU design faces bottlenecks including heavy reliance on expert knowledge, high manual labor costs, and scarcity of domain-specific training data. To address these challenges, this paper proposes the first LLM-based automated TPU generation framework, specifically targeting systolic array architectures and enabling end-to-end hardware synthesis—from high-level specifications to synthesizable RTL. Key contributions are: (1) the first open-source, high-quality hardware-specific dataset for training and evaluation; (2) a hardware-semantic-aware RAG mechanism that substantially mitigates LLM hallucination; and (3) integrated modeling of systolic arrays, optimized approximate multiply-accumulate units, and automated hardware pipelining. Experimental results demonstrate that the generated TPUs achieve, on average, 92% reduction in area and 96% reduction in power consumption compared to manually optimized baselines, with significant improvements in energy efficiency and overall PPA (performance–power–area).

Conventional TPU architectures exhibit low energy efficiency and high power consumption during inference of generative models (e.g., LLMs and diffusion Transformers). Method: This work proposes a novel TPU architecture integrating digital Compute-in-Memory (CIM) units directly into the matrix multiplication unit (MXU), replacing the traditional systolic array to fundamentally alleviate the von Neumann bottleneck. We further co-design a customized microarchitectural model and a hardware-aware mapping methodology tailored for generative models, enabling end-to-end optimization in simulation. Contribution/Results: Compared to the TPUv4i baseline, our architecture achieves 44.2% and 33.8% higher inference throughput for LLMs and diffusion Transformers, respectively, while reducing MXU energy consumption by 27.3×—yielding substantial improvements in both energy efficiency and computational throughput.

Traditional hardware utilization metrics fail to reflect true efficiency in large-scale ML fleets (e.g., Google TPU clusters) due to deep, multi-layered software–hardware coupling across models, data pipelines, frameworks, compilers, and schedulers. Method: This paper introduces an end-to-end cluster productivity analysis framework grounded in production-grade TPU telemetry, workload characterization, hierarchical performance attribution, and actionable optimization recommendations. Contribution/Results: We propose ML Productivity Goodput (MPG), the first holistic, stack-wide efficiency metric explicitly designed to quantify productive throughput—spanning models, datasets, frameworks, compilers, and schedulers—thereby transcending conventional hardware-centric utilization measures. Evaluated on real internal workloads, MPG enables precise, cross-layer bottleneck identification and drives measurable improvements in fleet effective throughput and resource return on investment.

This study addresses the GPU computational efficiency bottleneck in deep and machine learning. Methodologically, it proposes a task-aware GPU parallel architecture adaptation framework that systematically integrates CUDA stream-based concurrency, dynamic parallelism, and heterogeneous hardware (FPGA/TPU/ASIC) co-selection—implemented via deep integration into PyTorch, TensorFlow, and XGBoost. Its core contribution lies in establishing a transferable GPU optimization methodology, transcending model- or library-specific tuning. Experimental evaluation demonstrates 3–8× speedup across representative training and inference workloads. Furthermore, the authors open-source a modular, well-documented GPU optimization practice guide, substantially lowering the barrier to entry for AI practitioners seeking parallelization optimizations.

This work addresses the lack of an efficient lightweight machine learning runtime for AI inference on the VideoCore VII QPU of the Raspberry Pi 5. It presents the first end-to-end ML runtime stack tailored to this architecture, built upon the py-videocore7 assembly library. The implementation includes optimized kernels for tiled matrix multiplication, GEMM-based convolution, and single-head attention. A key innovation is the introduction of the smul24 instruction to enable integer-dense computation with INT16 inputs and INT32 accumulation, further enhanced by a persistent executor for improved efficiency. Experimental results demonstrate that the proposed dense operators achieve nearly two orders of magnitude speedup over NumPy and significantly outperform both PyTorch and NumPy across multiple workloads, underscoring the Raspberry Pi 5’s potential as an edge AI acceleration platform.

This work addresses the inefficiency of AI ASICs—such as TPUs—optimized for low-precision computation when executing high-precision number-theoretic transforms (NTTs) required by fully homomorphic encryption (FHE). Existing approaches suffer from precision mismatches that force NTT output reconstruction off the matrix engine’s dataflow, creating a performance bottleneck. To overcome this, the paper proposes a lightweight multi-precision systolic array architecture that, for the first time, seamlessly integrates low-precision matrix multiplication and high-precision NTT reconstruction within a unified dataflow, eliminating the need for vector processor intervention and avoiding dataflow disruption. Evaluated via matrix decomposition, shift-and-add reconstruction, 7nm OpenRoad synthesis, and SCALE-Sim simulation on a 128×128 matrix engine, the design achieves at least 1.33× speedup for NTT sizes ranging from 2¹² to 2¹⁶ with negligible hardware overhead, significantly enhancing FHE execution efficiency on standard AI ASICs.

This work addresses the architectural challenges posed by rapidly evolving deep neural networks such as Transformers by systematically designing and iteratively refining five generations of TPU-based training supercomputers. Maintaining architectural stability, the system progressively enhances scalability, fault tolerance, energy efficiency, and sustainability. Key innovations—including high-bandwidth memory (HBM), optical circuit switching, built-in self-test (BIST), and hardware replay—enable robust large-scale distributed training. The project identifies six critical characteristics for future training accelerators, achieving a 100-fold increase in per-node peak performance, a 3,600-fold improvement in overall system performance, and a 10-fold gain in HBM capacity and bandwidth. These advances significantly improve performance per watt and substantially reduce carbon emissions.

This study addresses the lack of reproducible end-to-end fine-tuning and deployment pipelines for Gemma-4 31B on TPUs, as well as the unclear performance and cost trade-offs between TPUs and GPUs for large language model training and inference. We present the first successful implementation of LoRA fine-tuning and inference serving for Gemma-4 31B on Google Cloud TPU, fully migrating an existing PyTorch GPU workflow to the JAX+Tunix/Qwix stack and introducing Orbax-to-safetensors conversion alongside vLLM-TPU Docker deployment. Experimental results demonstrate that, compared to an H100 GPU baseline, TPU v5p-8 achieves 1.61× faster training with 2.12× lower cost, while TPU v6e-8 delivers comparable inference throughput and 50% lower first-token latency. Overall, the combined training and serving cost is reduced by 1.82×, filling a critical gap in open-source TPU tooling and establishing a key technical pathway for cross-platform large model adaptation.

This work reveals that Cloud TPUs exhibit severe performance disadvantages—up to 4,693–6,908× slower—than GPUs for finite-field cryptographic computations, primarily due to the absence of wide-integer ALUs and extremely low spatial utilization (only 6.25% in the M dimension) of their matrix compute units. To address this, the authors propose a “spatial collapse” model that reformulates low-degree polynomial arithmetic into matrix-based Number Theoretic Transforms (NTT), integrated with Montgomery reduction for efficient finite-field operations. The study provides the first quantitative characterization of TPUs’ structural limitations in exact-domain computation and introduces a reproducible measurement framework grounded in HLO-level post-hoc validation, effectively circumventing interference from XLA fusion optimizations. This approach establishes a new paradigm for heterogeneous cryptographic computing.