PRIMAL: Processing-In-Memory Based Low-Rank Adaptation for LLM Inference Accelerator

2026-01-20

AI Summary

This work addresses the high energy consumption and low throughput bottlenecks of large language models during low-rank adaptation (LoRA) inference. The authors propose a processing-in-memory (PIM)-based heterogeneous acceleration architecture that integrates customized processing elements via a 2D mesh interconnect. By co-designing optimized spatial mapping, dataflow scheduling, and a novel synergistic mechanism combining SRAM reprogramming with power gating (SRPG), the architecture enables pipelined LoRA updates and sublinear power scaling. Evaluated on the Llama-13B model with LoRA rank 8, the proposed design achieves 1.5× higher throughput and 25× better energy efficiency compared to an NVIDIA H100 GPU.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This paper presents PRIMAL, a processing-in-memory (PIM) based large language model (LLM) inference accelerator with low-rank adaptation (LoRA). PRIMAL integrates heterogeneous PIM processing elements (PEs), interconnected by 2D-mesh inter-PE computational network (IPCN). A novel SRAM reprogramming and power gating (SRPG) scheme enables pipelined LoRA updates and sub-linear power scaling by overlapping reconfiguration with computation and gating idle resources. PRIMAL employs optimized spatial mapping and dataflow orchestration to minimize communication overhead, and achieves $1.5\times$ throughput and $25\times$ energy efficiency over NVIDIA H100 with LoRA rank 8 (Q,V) on Llama-13B.

Problem

• Processing-in-Memory
• Large Language Model
• Low-Rank Adaptation
• Inference Acceleration
• Hardware Efficiency

Innovation

• Processing-in-Memory
• Low-Rank Adaptation
• PIM Accelerator
• SRAM Reprogramming
• Energy Efficiency