Conversational Gold: Evaluating Personalized Conversational Search System using Gold Nuggets

2025-03-12

AI Summary

Existing automatic evaluation metrics for Retrieval-Augmented Generation (RAG) systems in personalized conversational search lack fine-grainedness and interpretability. Method: This paper introduces CONE-RAG, the first open-source benchmark jointly evaluating long-form answer generation and retrieval, built upon the TREC iKAT 2024 dataset. It incorporates 17-turn dialogues, 20,575 paragraph annotations, 2,279 gold nuggets, and 62 human-written answers, and proposes a “nugget extraction–matching–generation” evaluation framework. Contributions include: (1) a novel fine-grained gold-nugget-based evaluation paradigm; (2) explicit hallucination detection for unanswerable questions; (3) enhanced user profile modeling; and (4) reformulating the Passage-to-Knowledge Base (PTKB) task from ranking to classification. Results: Experiments demonstrate that CONE-RAG substantially improves the reliability and interpretability of automatic RAG evaluation.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Natural Language Processing: Question Answering
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Community question answering

Abstract

The rise of personalized conversational search systems has been driven by advancements in Large Language Models (LLMs), enabling these systems to retrieve and generate answers for complex information needs. However, the automatic evaluation of responses generated by Retrieval Augmented Generation (RAG) systems remains an understudied challenge. In this paper, we introduce a new resource for assessing the retrieval effectiveness and relevance of response generated by RAG systems, using a nugget-based evaluation framework. Built upon the foundation of TREC iKAT 2023, our dataset extends to the TREC iKAT 2024 collection, which includes 17 conversations and 20,575 relevance passage assessments, together with 2,279 extracted gold nuggets, and 62 manually written gold answers from NIST assessors. While maintaining the core structure of its predecessor, this new collection enables a deeper exploration of generation tasks in conversational settings. Key improvements in iKAT 2024 include: (1) ``gold nuggets'' -- concise, essential pieces of information extracted from relevant passages of the collection -- which serve as a foundation for automatic response evaluation; (2) manually written answers to provide a gold standard for response evaluation; (3) unanswerable questions to evaluate model hallucination; (4) expanded user personas, providing richer contextual grounding; and (5) a transition from Personal Text Knowledge Base (PTKB) ranking to PTKB classification and selection. Built on this resource, we provide a framework for long-form answer generation evaluation, involving nuggets extraction and nuggets matching, linked to retrieval. This establishes a solid resource for advancing research in personalized conversational search and long-form answer generation. Our resources are publicly available at https://github.com/irlabamsterdam/CONE-RAG.

Problem

• Evaluating retrieval effectiveness of RAG systems.
• Assessing relevance of responses using gold nuggets.
• Developing framework for long-form answer generation evaluation.

Innovation

• Uses nugget-based framework for response evaluation
• Introduces gold nuggets for automatic assessment
• Expands dataset with unanswerable questions and personas