CoRun: Padding is Simple and Efficient for Deterministic LLM Inference

2026-08-14

AI Summary

This study addresses output non-determinism caused by batch dependencies in LLM inference and the performance degradation of existing solutions by proposing CoRun. This system achieves deterministic inference through position invariance rather than batch invariance, isolating prefill from fixed-shape decode scheduling while leveraging CUDA graphs to optimize execution efficiency. Experimental results demonstrate that CoRun ensures strict result consistency while improving throughput by 15%–324% and reducing time-to-first-token and per-token latency by 51.8% and 48.6%, respectively. Significantly outperforming current batch-invariant kernel approaches, CoRun effectively reconciles inference determinism with high system performance, offering a robust solution for reliable and efficient large language model serving.

Abstract

Despite fixed sampling parameters and random seeds, Large Language Model (LLM) inference exhibits output inconsistency, which undermines downstream tasks such as model evaluation and reinforcement learning. A major source of this nondeterminism is batch-dependent GPU execution: dynamic input shapes change kernel tiling and floating-point reduction orders. Existing systems address this problem with batch-invariant kernels, but these kernels restrict optimized tiling and split reductions, increasing more than 2$\times$ latency and reducing serving throughput by up to 74 %. This paper observes that although most kernels are not batch-invariant, they are position-invariant. Leveraging this property, we present CoRun, a scheduling-based system that achieves deterministic inference without requiring batch invariance. CoRun employs isolated prefill and fixed-shape batched decode to handle the two stages of LLM inference, respectively, leveraging CUDA graphs for efficient execution and simplified implementation. Experiments on LLMs with diverse architectures, including Qwen and DeepSeek, show that CoRun ensures determinism while improving throughput by 15-324 % over batch-invariant approaches, reducing time-to-first-token by 51.8 % and time-per-output-token by 48.6 % on average.

Problem

• Deterministic LLM Inference
• Output Inconsistency
• Batch-dependent GPU Execution
• Nondeterminism
• Inference Throughput

Innovation

• Deterministic Inference
• Position-invariant Kernels
• CoRun
• Fixed-shape Batched Decode
• CUDA Graphs