Multi-Source Knowledge Pruning for Retrieval-Augmented Generation: A Benchmark and Empirical Study

2024-09-03

AI Summary

Existing RAG methods predominantly rely on single knowledge sources, rendering them inadequate for real-world scenarios requiring integration of heterogeneous, cross-domain knowledge—spanning structured (e.g., databases, tables) and unstructured (e.g., text, images) modalities—while lacking standardized benchmarks and systematic evaluation. Method: We formally define the multi-source knowledge RAG task and introduce PruningRAG, a novel framework featuring a three-stage, multi-granularity knowledge filtering mechanism: semantic similarity pruning, domain-adaptive filtering, and hierarchical attention distillation—dynamically suppressing noise and redundancy. Contribution/Results: We release the first open-source benchmark encompassing multimodal, cross-domain, structured, and unstructured knowledge. On this benchmark, PruningRAG achieves an average 12.7% improvement in QA accuracy and a 31.4% reduction in hallucination rate over state-of-the-art RAG baselines.

Technology Category

• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data
• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Knowledge Acquisition

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Retrieval-augmented generation (RAG) is increasingly recognized as an effective approach to mitigating the hallucination of large language models (LLMs) through the integration of external knowledge. While numerous efforts, most studies focus on a single type of external knowledge source. In contrast, most real-world applications involve diverse knowledge from various sources, a scenario that has been relatively underexplored. The main dilemma is the lack of a suitable dataset incorporating multiple knowledge sources and pre-exploration of the associated issues. To address these challenges, we standardize a benchmark dataset that combines structured and unstructured knowledge across diverse and complementary domains. Building upon the dataset, we identify the limitations of existing methods under such conditions. Therefore, we develop PruningRAG, a plug-and-play RAG framework that uses multi-granularity pruning strategies to more effectively incorporate relevant context and mitigate the negative impact of misleading information. Extensive experimental results demonstrate superior performance of PruningRAG and our insightful findings are also reported. Our dataset and code are publicly availablefootnote{https://github.com/USTCAGI/PruningRAG}.

Problem

• Multi-source knowledge integration in RAG
• Mitigating misinformation in language models
• Developing a benchmark for diverse knowledge sources

Innovation

• Multi-source knowledge integration
• Multi-granularity pruning strategies
• Standardized benchmark dataset