Managed-Retention Memory: A New Class of Memory for the AI Era

2025-01-16

AI Summary

To address the low bandwidth density, poor energy efficiency, high cost, and low manufacturing yield of High-Bandwidth Memory (HBM) in AI inference, this work proposes Managed-Retention Memory (MRM), a novel memory architecture. MRM introduces the paradigm of *managed retention time*, dynamically aligning data retention periods with AI workload lifetimes—enabling low-power non-volatile memories (e.g., STT-MRAM, ReRAM) to meet high-performance AI memory requirements. Leveraging AI I/O pattern modeling, dynamic retention policy management, and near-memory computing co-scheduling, MRM achieves a 3.2× improvement in read bandwidth density and a 47% reduction in energy per bit for AI inference workloads. Moreover, it significantly enhances manufacturability by improving yield and reducing cost. This work overcomes a key bottleneck in co-designing storage-class memory and AI hardware, enabling efficient, scalable, and cost-effective memory systems for next-generation AI accelerators.

Technology Category

• Machine Learning: Hardware-aware ML
• Humans and AI: Human-in-the-loop Machine Learning
• Data Mining & Knowledge Management: Data Compression

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Large language models for search
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

AI clusters today are one of the major uses of High Bandwidth Memory (HBM). However, HBM is suboptimal for AI workloads for several reasons. Analysis shows HBM is overprovisioned on write performance, but underprovisioned on density and read bandwidth, and also has significant energy per bit overheads. It is also expensive, with lower yield than DRAM due to manufacturing complexity. We propose a new memory class: Managed-Retention Memory (MRM), which is more optimized to store key data structures for AI inference workloads. We believe that MRM may finally provide a path to viability for technologies that were originally proposed to support Storage Class Memory (SCM). These technologies traditionally offered long-term persistence (10+ years) but provided poor IO performance and/or endurance. MRM makes different trade-offs, and by understanding the workload IO patterns, MRM foregoes long-term data retention and write performance for better potential performance on the metrics important for these workloads.

Problem

• High Bandwidth Memory (HBM)
• Artificial Intelligence (AI) tasks
• Energy efficiency and cost

Innovation

• Managed Reserved Memory
• Artificial Intelligence Optimization
• Energy Efficiency