Do Retrieval-Augmented Language Models Adapt to Varying User Needs?

2025-02-27

AI Summary

Existing RALM evaluation benchmarks overlook user需求 diversity, failing to characterize model adaptability across distinct usage intents. Method: We propose a user需求-driven, multi-dimensional evaluation framework centered on three canonical intents—Context-Exclusive, Context-First, and Memory-First—and introduce the synthetic URAQ dataset. We conduct controlled experiments on multi-task QA benchmarks (e.g., HotpotQA, DisentQA), systematically perturbing both retrieved evidence types (matching/conflicting/irrelevant) and instruction formulations. Contribution/Results: Our analysis reveals, for the first time, that architectural family—not fine-tuning strategy—primarily governs behavioral differences across intents. Constraining memory utilization improves robustness against adversarial retrieval but degrades peak performance under ideal retrieval conditions. These findings empirically validate the necessity and guiding value of user-centered evaluation for RALM design and optimization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recent advancements in Retrieval-Augmented Language Models (RALMs) have demonstrated their efficacy in knowledge-intensive tasks. However, existing evaluation benchmarks often assume a single optimal approach to leveraging retrieved information, failing to account for varying user needs. This paper introduces a novel evaluation framework that systematically assesses RALMs under three user need cases-Context-Exclusive, Context-First, and Memory-First-across three distinct context settings: Context Matching, Knowledge Conflict, and Information Irrelevant. By varying both user instructions and the nature of retrieved information, our approach captures the complexities of real-world applications where models must adapt to diverse user requirements. Through extensive experiments on multiple QA datasets, including HotpotQA, DisentQA, and our newly constructed synthetic URAQ dataset, we find that restricting memory usage improves robustness in adversarial retrieval conditions but decreases peak performance with ideal retrieval results and model family dominates behavioral differences. Our findings highlight the necessity of user-centric evaluations in the development of retrieval-augmented systems and provide insights into optimizing model performance across varied retrieval contexts. We will release our code and URAQ dataset upon acceptance of the paper.

Problem

• Assess RALMs' adaptability to user needs.
• Evaluate RALMs under diverse retrieval contexts.
• Optimize RALMs for varied user requirements.

Innovation

• Novel evaluation framework for RALMs
• Assesses RALMs under diverse user needs
• Restricts memory usage for robustness