Exploiting Primacy Effect To Improve Large Language Models

2025-07-18

AI Summary

This work identifies a significant primacy bias in large language models (LLMs) on multiple-choice question answering (MCQA), wherein models disproportionately favor earlier-listed answer options; critically, supervised fine-tuning exacerbates this bias. To address it, we propose an unsupervised, annotation-free option reordering method that dynamically reshuffles answer options based on their semantic similarity to the question—without modifying model parameters. Through fine-grained positional bias analysis, we provide the first empirical evidence that fine-tuning amplifies primacy bias, and we leverage this cognitive bias as an opportunity for performance improvement. Experiments across multiple MCQA benchmarks demonstrate consistent accuracy gains of 3.2–5.7 percentage points. Our approach establishes a novel paradigm for bias-aware model optimization, demonstrating that mitigating positional bias need not require architectural changes or labeled data.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Language Models (LLMs) have become essential in many Natural Language Processing (NLP) tasks, leveraging extensive pre-training and fine-tuning to achieve high accuracy. However, like humans, LLMs exhibit biases, particularly positional biases such as primacy and recency effects, which can influence the accuracy of the answers. The primacy effect-where items presented first are more likely to be remembered or selected-plays a key role in Multiple Choice Question Answering (MCQA), where the order of answer options can affect prediction outcomes. This study focuses on primacy bias in fine-tuned LLMs: We first show that fine-tuning amplifies this bias, probably due to exposure to human-like patterns. Hence, we strategically leverage this effect by reordering response options based on semantic similarity to the query, without requiring knowledge of the correct answer. Our experimental results show that this approach significantly improves performance in MCQA. More generally, our findings underscore the dual nature of biases as both challenges and opportunities, offering insights for bias-aware model design and NLP applications.

Problem

• Addressing primacy bias in fine-tuned LLMs for MCQA
• Leveraging semantic similarity to reorder answer options
• Improving model performance by exploiting positional biases

Innovation

• Leverage primacy bias via option reordering
• Use semantic similarity for answer ordering
• Improve MCQA without correct answer knowledge