Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs

2026-06-14

AI Summary

This study addresses a critical gap in the evaluation of large language models (LLMs)—namely, their answer stability when confronted with plausible counterarguments. The authors propose a controlled protocol that, for multiple-choice questions correctly answered by a model, generates coherent rebuttals targeting the incorrect options to test whether the model reverses its initial response. Introducing both self-attributed and cross-model rebuttal sources, they construct MaxFlip, a high-flip challenge set designed to probe model robustness under adversarial pressure. Evaluating seven state-of-the-art models across all 57 MMLU subjects, the experiments reveal flip rates ranging from 17.5% to 97.3%. Self-attributed rebuttals increase flip rates by an average of 7.1 percentage points, while MaxFlip further elevates them by up to 23.6 percentage points, highlighting significant vulnerabilities in current LLM reasoning consistency.

Abstract

Standard accuracy benchmarks are designed to test how closely large language models (LLMs) approach correct answers, but are not suitable for testing whether LLMs stick with a correct answer when that answer is challenged by a plausible counter-argument. We introduce a controlled protocol for evaluating answer stability: after a model answers a multiple-choice question correctly, we challenge the model's answer with a coherent argument for an incorrect option and measure whether the model flips. The setup a) isolates argumentative content from overt social pressure and b) varies argument length, self-attribution, and cross-model source. Across seven frontier models and 57 MMLU subjects, flip rates range from 17.5% to 97.3%, revealing large differences in stability that are not captured by accuracy metrics alone. We find that self-attribution consistently increases flip rates (mean +7.1pp, up to +18.7pp). Also, pooling wrong-answer arguments across models and selecting the most effective one per question yields stronger adversarial challenges than relying on any single source model. We further construct MaxFlip, a curated challenge set that amplifies flips by up to +23.6pp over standard self-generated challenges. We release the protocol, challenge records, and MaxFlip to support stability evaluation alongside standard accuracy benchmarks. Materials are available at https://github.com/nafisenik/WhoFlips and https://hf.co/datasets/nafisehNik/WhoFlips.

Problem

• answer stability
• large language models
• counterarguments
• flip rate
• adversarial challenges

Innovation

• answer stability
• counterargument evaluation
• self-attribution
• cross-model adversarial challenge
• MaxFlip