🤖 AI Summary
This study addresses the lack of a systematic taxonomy for failure modes in multi-provider large language model (LLM) service gateways, which hinders effective detection and diagnosis in production environments. The work proposes the first dual-axis structured failure classification framework for LLM gateways, categorizing failures by origin layer—spanning network/transport, streaming/protocol, state/session, model behavior, and governance/cost—and by detectability (explicit vs. implicit). Through root cause analysis, stress testing, mining of public bug reports, and protocol-level debugging, the authors construct a catalog of five validated failure cases, three of which include reproducible scripts. Notably, the study uncovers two previously undocumented silent failures: conversation history loss due to concurrency races and stream-index collisions corrupting tool-call payloads. These return HTTP 200 responses and pass standard health checks, evading detection without semantic-level observability, thereby posing significant threats to application reliability.
📝 Abstract
Multi-provider LLM gateways reverse proxies that route, load-balance, and rate-limit requests across foundation-model APIs have become critical production infrastructure. Yet the failure modes specific to this architectural layer remain undocumented, scattered across issue trackers and post-mortems with no unifying framework. We introduce \fa{}, a two-axis taxonomy that classifies failures by their \emph{origin layer} (Network/Transport, Streaming/Protocol, State/Session, Model~Behavior, Governance/Cost) and their \emph{detectability} (Loud vs.\ Silent). We populate this taxonomy with five verified catalog entries sourced from public bug reports and first-hand stress testing, each accompanied by a mechanistic root-cause analysis. Three entries include standalone reproduction scripts. Our principal finding is that the most operationally severe failures are \emph{silent}: they return HTTP~200, pass every standard health check, and corrupt application state in ways that require semantic-level observability to detect. Two such silent failures a concurrency race condition causing history loss and a streaming index collision corrupting tool-call payloads were discovered first-hand during \cb{} evaluation campaigns.