OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain

2024-12-17

arXiv.org

AI Summary

Existing RAG systems in finance lack comprehensive, automated, and trustworthy evaluation methodologies. Method: This paper introduces FinRAGE—the first fully automated, domain-specific RAG benchmark for finance—featuring a matrix-based RAG scenario taxonomy covering five task types and sixteen financial topics; a hybrid data construction paradigm integrating GPT-4 generation with human verification; a novel retrieval–generation joint evaluation framework; and a trustworthy metric system combining rule-based scoring with LLM-assisted assessment. Contributions/Results: Leveraging supervised fine-tuning of dedicated evaluators, financial knowledge modeling, and multi-granularity task–topic matrix design, FinRAGE achieves an 87.47% human acceptance rate on generated samples. The benchmark—including code and evaluation datasets—is publicly released. Empirical analysis reveals critical performance bottlenecks of RAG in vertical domains, providing a methodological foundation for trustworthy deployment of large language models in financial applications.

Technology Category

• Natural Language Processing: Generation
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

As a typical and practical application of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) techniques have gained extensive attention, particularly in vertical domains where LLMs may lack domain-specific knowledge. In this paper, we introduce an omnidirectional and automatic RAG benchmark, OmniEval, in the financial domain. Our benchmark is characterized by its multi-dimensional evaluation framework, including (1) a matrix-based RAG scenario evaluation system that categorizes queries into five task classes and 16 financial topics, leading to a structured assessment of diverse query scenarios; (2) a multi-dimensional evaluation data generation approach, which combines GPT-4-based automatic generation and human annotation, achieving an 87.47% acceptance ratio in human evaluations on generated instances; (3) a multi-stage evaluation system that evaluates both retrieval and generation performance, result in a comprehensive evaluation on the RAG pipeline; and (4) robust evaluation metrics derived from rule-based and LLM-based ones, enhancing the reliability of assessments through manual annotations and supervised fine-tuning of an LLM evaluator. Our experiments demonstrate the comprehensiveness of OmniEval, which includes extensive test datasets and highlights the performance variations of RAG systems across diverse topics and tasks, revealing significant opportunities for RAG models to improve their capabilities in vertical domains. We open source the code of our benchmark in href{https://github.com/RUC-NLPIR/OmniEval}{https://github.com/RUC-NLPIR/OmniEval}.

Problem

• Evaluate RAG systems in finance
• Develop multi-dimensional RAG evaluation
• Enhance RAG reliability with robust metrics

Innovation

• Matrix-based RAG scenario evaluation system
• Multi-dimensional evaluation data generation
• Multi-stage evaluation system for RAG