Where is the answer? Investigating Positional Bias in Language Model Knowledge Extraction

2024-02-16

AI Summary

This paper identifies a “positional bias” in fine-tuning large language models (LLMs): while models accurately extract information from document initial sentences, their performance degrades significantly for mid- and end-position knowledge—a deficiency rooted in the inherent position dependence of autoregressive training (“perplexity curse”). To address this, the authors conduct the first systematic mechanistic analysis of the bias and propose Denoising Autoregressive Regularization (DAR), a novel training technique that explicitly decouples positional and content modeling. They further construct the first benchmark dataset—combining synthetic and real-world examples—that supports position-aware question answering evaluation. Experiments across model scales confirm pervasive positional bias; DAR improves answer extraction accuracy at mid- and end-positions by up to 37%. These findings establish new theoretical foundations and practical tools for co-designing retrieval-augmented generation (RAG) and LLM fine-tuning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Large language models require updates to remain up-to-date or adapt to new domains by fine-tuning them with new documents. One key is memorizing the latest information in a way that the memorized information is extractable with a query prompt. However, LLMs suffer from a phenomenon called perplexity curse; despite minimizing document perplexity during fine-tuning, LLMs struggle to extract information through a prompt sentence. In this new knowledge acquisition and extraction, we find a very intriguing fact that LLMs can accurately answer questions about the first sentence, but they struggle to extract information described in the middle or end of the documents used for fine-tuning. Our study suggests that the auto-regressive training causes this issue; each token is prompted by reliance on all previous tokens, which hinders the model from recalling information from training documents by question prompts. To conduct the in-depth study, we publish both synthetic and real datasets, enabling the evaluation of the QA performance w.r.t. the position of the corresponding answer in a document. Our investigation shows that even a large model suffers from the perplexity curse, but regularization such as denoising auto-regressive loss can enhance the information extraction from diverse positions. These findings will be (i) a key to improving knowledge extraction from LLMs and (ii) new elements to discuss the trade-off between RAG and fine-tuning in adapting LLMs to a new domain.

Problem

• Investigates positional bias in LLM knowledge extraction.
• Examines auto-regressive training's impact on information recall.
• Proposes regularization to improve multi-position information extraction.

Innovation

• Denoising auto-regressive loss enhances information extraction
• Synthetic and real datasets evaluate QA performance
• Regularization mitigates positional bias in knowledge extraction