Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models

2024-06-03

AI Summary

Designing hardware platforms for distributed inference of next-generation large language models (LLMs) poses significant challenges due to architectural and optimization heterogeneity. This paper introduces GenZ, the first systematic software-hardware co-modeling framework for LLM inference. GenZ unifies analytical modeling of diverse architectures—including Dense, GQA, MoE, and Mamba—as well as key optimizations such as chunking, speculative decoding, and quantization. Leveraging analytical modeling, cycle-accurate simulation, and real-system calibration, it achieves end-to-end inference latency prediction with a geometric mean error of only 5.82%. GenZ precisely identifies dominant bottlenecks—computation, memory capacity/bandwidth, or network latency/bandwidth—under varying service-level objective (SLO) constraints, and enables multi-dimensional sensitivity analysis across architectures and optimization strategies. The framework is open-sourced and accompanied by an interactive web-based analysis platform.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) have shown remarkable performance across a wide range of applications, often outperforming human experts. However, deploying these gigantic models efficiently for diverse inference use cases requires carefully designed hardware platforms with ample computing, memory, and network resources. With constant innovation in LLM serving optimizations and model architecture evolving at breakneck speed, the hardware requirements to meet Service Level Objectives (SLOs) remain an open research question. To answer the question, we present an analytical tool, GenZ, to efficiently navigate the relationship between diverse LLM model architectures(Dense, GQA, MoE, Mamba), LLM serving optimizations(Chunking, Speculative decoding, quanitization), and AI platform design parameters. Our tool estimates LLM inference performance metrics for the given scenario. We have validated against real hardware platforms running various different LLM models, achieving a max geomean error of 5.82.We use GenZ to identify compute, memory capacity, memory bandwidth, network latency, and network bandwidth requirements across diverse LLM inference use cases. We also study diverse architectural choices in use today (inspired by LLM serving platforms from several vendors) to help inform computer architects designing next-generation AI hardware accelerators and platforms. The trends and insights derived from GenZ can guide AI engineers deploying LLMs as well as computer architects designing next-generation hardware accelerators and platforms. Ultimately, this work sheds light on the platform design considerations for unlocking the full potential of large language models across a spectrum of applications. The source code is available at https://github.com/abhibambhaniya/GenZ-LLM-Analyzer . Users can also be tried it on at https://genz-llm-analyzer.streamlit.app/ without any setup on your web browser.

Problem

• Efficient hardware platform design for next-gen LLM inference
• Balancing LLM architectures and serving optimizations with platform parameters
• Determining compute, memory, and network requirements for diverse LLM use cases

Innovation

• Analytical tool GenZ for LLM performance estimation
• Validates diverse LLM models on real hardware
• Guides AI platform design and optimization