Google's Training Supercomputers from TPU v2 to Ironwood: Architectural Stability, Scale, Resilience, Power Efficiency, and Sustainability Across Five Generations

2026-06-14

AI Summary

This work addresses the architectural challenges posed by rapidly evolving deep neural networks such as Transformers by systematically designing and iteratively refining five generations of TPU-based training supercomputers. Maintaining architectural stability, the system progressively enhances scalability, fault tolerance, energy efficiency, and sustainability. Key innovations—including high-bandwidth memory (HBM), optical circuit switching, built-in self-test (BIST), and hardware replay—enable robust large-scale distributed training. The project identifies six critical characteristics for future training accelerators, achieving a 100-fold increase in per-node peak performance, a 3,600-fold improvement in overall system performance, and a 10-fold gain in HBM capacity and bandwidth. These advances significantly improve performance per watt and substantially reduce carbon emissions.

Abstract

This paper (to appear in the July/August 2026 issue of IEEE Micro magazine) summarizes five generations of Google s TPUs, from TPU v2 to Ironwood, highlighting their evolution as scalable, resilient, power-efficient, sustainable supercomputers for AI training. It details the TPU s stable architecture, which has surprisingly easily accommodated the rapidly changing deep neural network workloads, such as the rise of Transformers. Key advancements over eight years include 10x increase in HBM capacity and bandwidth per node, a 100x increase in peak node performance, and a 3600x increase in supercomputer performance. The paper also discusses the role of optical circuit switches, built-in self test, and hardware replay in enhancing resilience and how TPU's environmental impact is reduced with substantial improvements in performance per Watt and in carbon emissions per floating point operation. It concludes by identifying six features that may well characterize successful training accelerators of this decade.

Problem

• AI training supercomputers
• architectural stability
• scalability
• power efficiency
• sustainability

Innovation

• architectural stability
• optical circuit switches
• hardware replay
• power efficiency
• sustainable computing