Inference economics of language models

2025-06-05

AI Summary

This work addresses the cost–latency trade-off in large language model (LLM) inference at scale by introducing the first systematic economic analysis framework. Methodologically, it jointly models computational cost, memory and network bandwidth bottlenecks, and end-to-end latency constraints to derive the Pareto frontier between per-token cost and generation throughput across hardware configurations. The framework integrates arithmetic intensity analysis, bandwidth-aware parallelism modeling, and latency-sensitive batch-size optimization. Empirical evaluation across multiple state-of-the-art LLMs reveals universal principles governing optimal parallelization strategies and batch sizes under hardware constraints. The results provide quantifiable, decision-ready guidance for industrial LLM deployment—enabling optimal hardware selection and scheduling configuration under given budget and latency requirements.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We develop a theoretical model that addresses the economic trade-off between cost per token versus serial token generation speed when deploying LLMs for inference at scale. Our model takes into account arithmetic, memory bandwidth, network bandwidth and latency constraints; and optimizes over different parallelism setups and batch sizes to find the ones that optimize serial inference speed at a given cost per token. We use the model to compute Pareto frontiers of serial speed versus cost per token for popular language models.

Problem

• Economic trade-off between cost and speed in LLM inference
• Optimizing parallelism and batch sizes for efficient deployment
• Computing Pareto frontiers for speed versus cost per token

Innovation

• Optimizes token cost and speed trade-off
• Considers arithmetic and bandwidth constraints
• Computes Pareto frontiers for models