🤖 AI Summary
This study investigates attribution reliability in Retrieval-Augmented Generation (RAG) for Long-Form Question Answering (LFQA)—specifically, how accurately and verifiably large language models (LLMs) ground answers in retrieved evidence. To this end, we construct the first human-annotated attribution dataset tailored to LFQA, enabling systematic analysis of answer variations across multiple LLMs conditioned on identical retrieval outputs. We quantify how retrieval quality affects answer fluency, length, and attribution accuracy. Further, we propose a semantic-alignment-based automatic attribution evaluation framework that identifies three canonical attribution errors—omission, misattribution, and fabrication—and their underlying causes. Experimental results demonstrate that retrieval quality strongly influences attribution correctness, and that attribution capability varies significantly across LLMs. Our work provides an empirical foundation and methodological toolkit for enhancing interpretability and trustworthiness assessment in RAG systems.
📝 Abstract
We present a study of retrieval-augmented language models (LMs) on long-form question answering. We analyze how retrieval augmentation impacts different LMs, by comparing answers generated from models while using the same evidence documents, and how differing quality of retrieval document set impacts the answers generated from the same LM. We study various attributes of generated answers (e.g., fluency, length, variance) with an emphasis on the attribution of generated long-form answers to in-context evidence documents. We collect human annotations of answer attribution and evaluate methods for automatically judging attribution. Our study provides new insights on how retrieval augmentation impacts long, knowledge-rich text generation of LMs. We further identify attribution patterns for long text generation and analyze the main culprits of attribution errors. Together, our analysis reveals how retrieval augmentation impacts long knowledge-rich text generation and provide directions for future work.
