VeritasFi: An Adaptable, Multi-tiered RAG Framework for Multi-modal Financial Question Answering

2025-10-12

AI Summary

Financial RAG systems face two key challenges: (1) difficulty in processing heterogeneous data (text, tables, charts), and (2) the trade-off between generalizability and domain-specific customization. To address these, we propose a multi-level Retrieval-Augmented Generation framework: (1) a unified multimodal preprocessing pipeline; (2) a tripartite hybrid retrieval engine integrating semantic search, real-time tool invocation, and expert memory retrieval; and (3) a two-stage contrastive learning re-ranking model enabling both generic pretraining and rapid fine-tuning on enterprise-specific data. Experiments demonstrate significant improvements in financial question answering across accuracy, multimodal coverage, and response latency. In public disclosure document scenarios, our framework achieves an effective balance between broad-domain capability and company-specific adaptation, while exhibiting strong scalability and deployment flexibility.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Retrieval-Augmented Generation (RAG) is becoming increasingly essential for Question Answering (QA) in the financial sector, where accurate and contextually grounded insights from complex public disclosures are crucial. However, existing financial RAG systems face two significant challenges: (1) they struggle to process heterogeneous data formats, such as text, tables, and figures; and (2) they encounter difficulties in balancing general-domain applicability with company-specific adaptation. To overcome these challenges, we present VeritasFi, an innovative hybrid RAG framework that incorporates a multi-modal preprocessing pipeline alongside a cutting-edge two-stage training strategy for its re-ranking component. VeritasFi enhances financial QA through three key innovations: (1) A multi-modal preprocessing pipeline that seamlessly transforms heterogeneous data into a coherent, machine-readable format. (2) A tripartite hybrid retrieval engine that operates in parallel, combining deep multi-path retrieval over a semantically indexed document corpus, real-time data acquisition through tool utilization, and an expert-curated memory bank for high-frequency questions, ensuring comprehensive scope, accuracy, and efficiency. (3) A two-stage training strategy for the document re-ranker, which initially constructs a general, domain-specific model using anonymized data, followed by rapid fine-tuning on company-specific data for targeted applications. By integrating our proposed designs, VeritasFi presents a groundbreaking framework that greatly enhances the adaptability and robustness of financial RAG systems, providing a scalable solution for both general-domain and company-specific QA tasks. Code accompanying this work is available at https://github.com/simplew4y/VeritasFi.git.

Problem

• Processes heterogeneous financial data formats like text and tables
• Balances general-domain applicability with company-specific adaptation
• Enhances accuracy and adaptability of financial question answering systems

Innovation

• Multi-modal preprocessing pipeline for heterogeneous data
• Tripartite hybrid retrieval engine for comprehensive scope
• Two-stage training strategy for domain adaptation