🤖 AI Summary
This study addresses the performance degradation caused by context mismatch when switching large language models (LLMs) in multi-turn dialogues—a phenomenon termed "silent drift." The work presents the first systematic characterization of this issue and introduces a benchmark evaluation framework based on a switching matrix. By employing pairwise turn-level bootstrap confidence intervals, the authors quantify the impact of model switching on dialogue performance across datasets such as CoQA and Multi-IF. A key innovation lies in decomposing the drift into prefix influence and suffix sensitivity, revealing systematic robustness or vulnerability of models to non-self-generated context. Experiments demonstrate that a single switch can alter Multi-IF’s strict success rate by −8 to +13 percentage points and induce CoQA F1 fluctuations of up to ±4 points, offering both theoretical insights and practical tools for effective risk monitoring.
📝 Abstract
Deployed multi-turn LLM systems routinely switch models mid-interaction due to upgrades, cross-provider routing, and fallbacks. Such handoffs create a context mismatch: the model generating later turns must condition on a dialogue prefix authored by a different model, potentially inducing silent performance drift. We introduce a switch-matrix benchmark that measures this effect by running a prefix model for early turns and a suffix model for the final turn, and comparing against the no-switch baseline using paired episode-level bootstrap confidence intervals. Across CoQA conversational QA and Multi-IF benchmarks, even a single-turn handoff yields prevalent and statistically significant, directional effects and may swing outcomes by -8 to +13 percentage points in Multi-IF strict success rate and +/- 4 absolute F1 on CoQA, comparable to the no-switch gap between common model tiers (e.g., GPT-5-nano vs GPT-5-mini). We further find systematic compatibility patterns: some suffix models degrade under nearly any non-self dialogue history, while others improve under nearly any foreign prefix. To enable compressed handoff risk monitoring, we decompose switch-induced drift into per-model prefix influence and suffix susceptibility terms, accounting for ~70% of variance across benchmarks. These results position handoff robustness as an operational reliability dimension that single-model benchmarks miss, motivating explicit monitoring and handoff-aware mitigation in multi-turn systems.