Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators

2026-06-14

AI Summary

This study systematically evaluates the practical performance advantages of emerging AI accelerators over GPUs in large language model (LLM) inference by decoupling the process into Prefill and Decode phases—a first in the field. Using the Llama2-7B model, the authors conduct phase-specific benchmarking and heterogeneous deployment experiments on platforms including GPU and GroqRack, measuring Time-To-First-Token (TTFT) and Time-Per-Output-Token (TPOT). Results show that GPUs consistently outperform in the Prefill phase, while GroqRack achieves lower TPOT in the Decode phase under no-batching conditions; however, GPUs surpass GroqRack in throughput under high batching. The work delineates the effective conditions for heterogeneous task partitioning, offering actionable insights for hardware selection and scheduling in LLM inference systems.

Abstract

As large language models (LLMs) are increasingly deployed in latency- and cost-sensitive settings, inference efficiency has become a central systems challenge. While GPUs dominate current deployments, a growing number of AI accelerators claim advantages for LLM inference, yet it remains unclear under which conditions such accelerators outperform GPUs in practice. Recent inference systems decompose execution into Prefill and Decode phases, which exhibit distinct computational characteristics and latency metrics, commonly captured by time to first token (TTFT) and time per output token (TPOT). This paper presents a phase-aware evaluation of LLM inference performance across GPUs and emerging AI accelerators using a common model, Llama2-7B. By separately measuring Prefill and Decode performance, we reveal that accelerator advantages differ by phase and metric. Our results show that GPUs consistently excel in the compute-intensive Prefill phase, while GroqRack achieves significantly lower TPOT during Decode (batching not currently supported). However, GPUs regain an advantage in Decode throughput as batch size increases. These findings demonstrate that each platform exhibits distinct phase-dependent strengths. We further analyze heterogeneous Prefill/Decode disaggregation across different accelerator platforms, identifying performance gains and the workload and network conditions under which such gains are realized.

Problem

• LLM inference
• AI accelerators
• Prefill/Decode
• performance evaluation
• latency metrics

Innovation

• Prefill/Decode-aware evaluation
• LLM inference
• AI accelerators
• phase-dependent performance
• heterogeneous disaggregation