Jagged Judges: Epistemic Stability Under Silence, Pressure, and Persistence

2026-08-12

AI Summary

This work addresses the cognitive instability of large language models (LLMs) when acting as evaluators under repeated questioning or adversarial pressure, a vulnerability not adequately captured by accuracy alone. To systematically assess this issue, the authors propose Wiggle, the first unified framework for stress-testing cognitive stability in evaluation tasks. Evaluating nine state-of-the-art models across 14 tasks along three dimensions—mechanical consistency, single-turn belief steadfastness, and multi-turn persistence—the study employs adversarial re-prompting, single-round challenges, and multi-round adaptive persuasion strategies. Using jury-majority vote strength as a predictive signal, results reveal judgment flip rates of 25%–71% under static pressure and 62%–91% under adversarial persuasion. Notably, most successful perturbations that altered judgments actually reduced alignment with ground-truth labels, confirming jury-majority vote as the strongest single-instance prediction signal.

Abstract

LLM judges have become central infrastructure for model evaluations, online grading, and reward modeling. Judges are typically validated by accuracy on golden data, but accuracy says little about whether they are stable under re-prompting, challenge, or sustained pushback. We introduce the \emph{Wiggle Framework}, a unified stress test for epistemic stability in LLM judges. The framework decomposes judge robustness along three dimensions: Mechanical Consistency (stability under re-prompting and reframing), Single-turn Conviction (stability under a single challenge), and Multi-turn Persistence (stability under sustained or adaptive pressure). We use the framework to study 9 frontier models across 14 judging tasks spanning safety, toxicity, AI writing detection, and political-response evaluation. Every model exhibits substantial wiggle as a judge --- flipping verdicts 25--71\% of the time under static pushback, and 62--91\% with an adversarial LLM persuader. Critically, we find that pressure that succeeds in changing a judge's verdict is almost always net-corrupting with respect to ground truth. Beyond the framework itself, we identify baseline jury majority strength as the most effective single-shot signal for anticipating which items wiggle. Taken together, this is the first apples-to-apples cross-dataset comparison of mechanical, conformity, and persuadability tests in a judging context.

Problem

• epistemic stability
• LLM judges
• robustness
• judgment consistency
• adversarial pressure

Innovation

• Wiggle Framework
• epistemic stability
• LLM judges
• robustness evaluation
• adversarial persuasion