The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context

2026-07-14

AI Summary

This study addresses the limitations of aggregate accuracy metrics in capturing prediction instability at the individual sample level when large language models process inputs containing task-irrelevant context. By systematically inserting random-character pseudowords as irrelevant context prefixes, the authors evaluate how predictions vary across different models, datasets, context lengths, and inference budgets. They find that while such irrelevant context exerts minimal impact on overall performance, it induces significant prediction flips on a subset of samples—termed fragile instances—which exhibit model-specific characteristics. The work highlights that conventional aggregate metrics obscure context-induced tail risks and underscores the necessity of evaluating per-sample reliability rather than relying solely on average performance measures.

Abstract

As large language models (LLMs) grow more capable, they are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context. In a controlled setting, we find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy. This aggregate stability, however, masks significant per-example instability. Even semantically meaningless pseudo-words, formed by randomly combining characters, can markedly shift model predictions on a small fraction of examples, degrading performance on some while improving it on others. This two-sided effect holds consistently across a wide range of models and datasets, yet the affected examples are largely model-specific. We further show that this instability is modulated by context type, context length, test-time compute, and model development stage. Together, our findings reveal context-induced tail risks concealed by aggregate accuracy, motivating per-example reliability evaluation of language models.

Problem

• prediction flips
• task-irrelevant context
• aggregate accuracy
• model robustness
• per-example instability

Innovation

• prediction flips
• task-irrelevant context
• aggregate accuracy
• per-example reliability
• tail risks