Speed and Conversational Large Language Models: Not All Is About Tokens per Second

2024-08-01

Computer

AI Summary

Existing LLM inference speed benchmarks—typically reported in tokens/second—lack task awareness and fail to reflect real-world deployment latency, obscuring critical dependencies on generation length, context size, and interaction patterns. Method: We propose the first task-aware LLM inference speed evaluation framework, decoupling these factors via CUDA profiling, dynamic batching analysis, and realistic dialogue trace replay across multiple backends (Llama.cpp, vLLM, HuggingFace Transformers). Contribution/Results: Empirical evaluation reveals up to 3.2× end-to-end latency variation for the same model across dialogue vs. summarization tasks. Key bottlenecks are identified as inefficient KV cache management and imbalance between prefill and decode phases. Our framework enables quantitative performance attribution, providing actionable insights for optimizing LLM deployment efficiency on GPU hardware.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The speed of open-weights large language models (LLMs) and its dependency on the task at hand, when run on GPUs, is studied to present a comparative analysis of the speed of the most popular open LLMs.

Problem

• Analyze speed of open-weights LLMs
• Dependency on tasks and GPUs
• Compare popular open LLMs speed

Innovation

• Open-weights LLMs speed analysis
• GPU performance dependency study
• Comparative speed evaluation methodology