Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models

2026-06-24

AI Summary

This study addresses the lack of permutation invariance in multimodal large language models (MLLMs), which exhibit inconsistent outputs under input ordering changes. The authors systematically formalize and quantify this issue by introducing a five-dimensional perturbation auditing framework encompassing options, evidence chunks, documents, images, and mixed modalities. Leveraging a Bayesian item response model, they disentangle order-induced noise from inherent dimensional biases, while employing same-order control groups to account for decoding stochasticity. The work proposes cross-order flip rate as a standardized evaluation metric and finds that all 18 state-of-the-art MLLMs violate permutation invariance, with average flip rates ranging from 24% to 50% and even the best-performing model exhibiting a 13.4% flip rate. Prompt engineering shows only limited, modality-specific efficacy and fails to generalize across settings.

Abstract

Standard benchmarks for multimodal large language models (MLLMs) score each item on one canonical ordering and miss whether order-irrelevant shuffling changes the answer, a baseline reliability property called for by emerging AI evaluation guidelines. We introduce Facet-Probe, a five-facet audit (option, evidence-chunk, document-rank, image-set, and mixed-modality ordering) of 18 frontier and open-weight MLLMs. A Bayesian item-response model separates ordering noise from per-facet bias, and a same-ordering control estimates the decoder-stochastic floor for observed flips. We find that none of the 18 MLLMs we audit are order-invariant: screened per-facet panel-mean flip rates span 24-50%. A Gemini same-ordering control at temperature 0 estimates a substantial ordering excess over a same-input decoder-noise floor in verified cells. Capability predicts but does not eliminate flips; the best model still flips on 13.4% of trials. In our Gemini mitigation tests, training-free prompt changes are modality-conditional and do not transfer from text to visual reasoning. These results suggest that prompt-level mitigation alone is unlikely to provide general order robustness, motivating future work on training-time and architectural approaches. We propose cross-ordering flip rate as a standard reporting axis for MLLMs.

Problem

• order sensitivity
• multimodal large language models
• input ordering
• reliability
• answer consistency

Innovation

• order sensitivity
• multimodal large language models
• Facet-Probe
• input invariance
• audit framework