Demystifying Numerical Instability in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEAL

2026-06-18

AI Summary

This work addresses the irreproducibility of large language model (LLM) inference on heterogeneous GPUs at 16-bit precision, a critical issue stemming from truncation errors that undermines reliability in high-stakes domains such as finance and healthcare. Through SASS-level instruction analysis, the authors identify the root cause of numerical instability and propose HEAL, a novel approach that synergistically combines INT16 quantization with an algebraic error compensation mechanism. HEAL achieves near-FP32 reproducibility without expanding KV cache memory or compromising Tensor Core throughput. Evaluated on the authors’ MCR-Bench benchmark, HEAL matches FP32 baseline reproducibility on critical tasks while reducing performance overhead by up to 7.1×.

Abstract

As Large Language Models (LLMs) deploy into mission-critical domains (e.g., finance, medicine, and law), output reproducibility has become a strict system requirement. While practitioners use greedy decoding to eliminate algorithmic stochasticity, empirical deployments with 16-bit precisions still exhibit catastrophic output divergence across heterogeneous GPUs. Through SASS-level profiling, we reveal that this inconsistency is fundamentally driven by truncation errors introduced during downcasting at kernel boundaries. However, achieving reproducibility via a global FP32 pipeline incurs prohibitive system penalties: bypassing 16-bit hardware accelerators hurts compute efficiency, while upcasting the KV cache doubles memory overhead. To bridge this gap, we propose Hybrid Error ALleviation (HEAL), a targeted intervention that approximates FP32 precision while resolving hardware constraints through two targeted mechanisms. First, recognizing that floating-point formats underutilize their bit-width for Q, K, V tensors, HEAL applies INT16 quantization that preserves numerical stability without expanding the KV cache footprint. Second, HEAL synthesizes high-precision matrix multiplications via an algebraic error compensation strategy, executing entirely on high-throughput 16-bit Tensor Cores. To evaluate our approach practically, we introduce MCR-Bench, a benchmark targeting reproducibility in mission-critical tasks. HEAL achieves the same level of reproducibility on downstream tasks as the FP32 baseline while reducing the performance overhead by up to 7.1x.

Problem

• numerical instability
• reproducibility
• LLM inference
• precision
• mission-critical tasks

Innovation

• numerical reproducibility
• HEAL
• mixed-precision inference
• tensor quantization
• error compensation