AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMs

2026-08-14

AI Summary

This study addresses the limitations of single-path evaluation and neglected anchor relevance in assessing anchoring effects within large language models. We construct a multi-path anchoring benchmark incorporating an explicit relevance dimension and conduct large-scale controlled experiments across fourteen models. Our results reveal the path-dependency of anchoring bias, demonstrating that plausible anchors induce more significant deviations and that high-accuracy models remain vulnerable to such semantically relevant anchors. By transcending traditional single-path evaluation paradigms, this work systematically elucidates the mechanisms underlying model bias across diverse reasoning paths and anchor semantics. Ultimately, these findings establish a novel framework for evaluating the cognitive robustness of large language models, highlighting critical vulnerabilities even in high-performing systems when exposed to contextually plausible misinformation.

Abstract

The anchoring effect is a cognitive bias in which an initial reference value shifts a later judgment toward itself. This effect is well established in human judgment and decision-making, and recent work suggests that large language models (LLMs) exhibit similar behavior. However, existing work on anchoring in LLMs typically evaluates only a narrow set of anchor pathways and rarely distinguishes irrelevant from plausible anchors. We introduce AnchorBench, a benchmark for the anchoring effect in LLMs that evaluates multiple anchor pathways under an explicit anchor relevance axis. Across fourteen models, including ten open-weight models and four frontier API models, and a large set of controlled prompts, we find that (1) anchoring is strongly pathway-dependent, (2) plausible anchors usually induce larger shifts than irrelevant ones when introduced through stronger pathways, (3) anchor influence generally weakens as the anchor moves farther from the evidence-supported answer, most clearly on External and RAG, and (4) high task accuracy on the anchor-free control condition (Acc$_{10}$: answers within 10 points of gold) does not guarantee robustness: even frontier API models above 95% control accuracy remain susceptible to plausible anchors.

Problem

• Anchoring Effect
• Large Language Models
• Cognitive Bias
• Anchor Pathways
• Benchmark

Innovation

• AnchorBench
• Anchoring Effect
• Anchor Relevance
• Multi-Pathway Evaluation
• LLM Robustness