Most LLM Conformity Needs No Speaker: Measuring the Speaker-Free Floor in Peer-Pressure Benchmarks

2026-07-06

AI Summary

This study addresses a critical confound in current evaluations of conformity in large language models (LLMs), where the presence of speakers and repetition of incorrect answers are conflated, obscuring the distinct effects of social pressure versus textual repetition. To disentangle these factors, the work introduces a novel “speaker-free floor” baseline, systematically assessing LLMs’ sensitivity to repeated answers absent social cues through controlled experiments across six open-source models and seven question-answering and reasoning datasets. The results reveal that 66.5% of initially correct responses are erroneously revised under the speaker-free condition—substantially higher than the 10.3% error rate from simple re-prompting—and that such revisions are made with high confidence and resist calibration. These findings demonstrate that much of the observed conformity stems from sensitivity to answer repetition rather than genuine social influence, prompting a reconceptualization of conformity evaluation methodologies.

Abstract

LLM conformity is often used to describe cases where a model changes a correct answer toward a peer or group response. We show that most of this apparent conformity survives even after the peer is removed. The reason is a confound: standard conformity prompts mix two cues at once, the presence of a speaker and the repeated wrong answer itself. Existing benchmarks vary these cues together, so they cannot tell how much of the revision actually depends on the speaker. We introduce a no-source condition: the same asserted answer with the explicit speaker removed. Across six open-weight LLMs and seven QA and reasoning datasets, this condition alone causes harmful revision in $66.5\%$ of initially correct cases, compared with $10.3\%$ under a plain re-ask. The effect also remains when the repeated answer is paraphrased and when answer options are hidden in an open-ended setting. Source framing mainly modulates this floor: expert-panel framing raises it, while minimal person labels do not reliably raise it. When models flip, they are usually confidently wrong, and simple recalibration does not recover the original answer. Source attribution still matters, but it should be measured as an increment above this speaker-free floor. The methodological lesson is that conformity benchmarks should first measure what remains after the speaker is removed; without this step, benchmarks may mistake repeated text for social influence.

Problem

• LLM conformity
• social influence
• benchmark confounding
• speaker-free effect
• answer repetition

Innovation

• speaker-free floor
• LLM conformity
• no-source condition
• social influence
• benchmark confound