🤖 AI Summary
This work addresses the often-overlooked output variability in large language models (LLMs) even under nominal temperature settings of \( T = 0 \), which undermines reproducibility and evaluation reliability due to non-determinism inherent in inference implementations. We introduce, for the first time, the concept of “background temperature” (\( T_{\text{bg}} \)) to formally characterize the implicit randomness introduced by the inference environment and establish a mapping between this effect and the equivalent temperature of an idealized reference system. Through stochastic perturbation modeling, an equivalent temperature estimation protocol, and cross-platform experimental validation, we empirically confirm the existence of \( T_{\text{bg}} \) across mainstream LLMs and demonstrate its tangible impact on model deployment and evaluation. This study provides the first quantifiable and measurable theoretical framework for understanding and analyzing inference-induced non-determinism in LLMs.
📝 Abstract
Even when decoding with temperature $T=0$, large language models (LLMs) can produce divergent outputs for identical inputs. Recent work by Thinking Machines Lab highlights implementation-level sources of nondeterminism, including batch-size variation, kernel non-invariance, and floating-point non-associativity. In this short note we formalize this behavior by introducing the notion of \emph{background temperature} $T_{\mathrm{bg}}$, the effective temperature induced by an implementation-dependent perturbation process observed even when nominal $T=0$. We provide clean definitions, show how $T_{\mathrm{bg}}$ relates to a stochastic perturbation governed by the inference environment $I$, and propose an empirical protocol to estimate $T_{bg}$ via the equivalent temperature $T_n(I)$ of an ideal reference system. We conclude with a set of pilot experiments run on a representative pool from the major LLM providers that demonstrate the idea and outline implications for reproducibility, evaluation, and deployment.