eSapiens: A Real-World NLP Framework for Multimodal Document Understanding and Enterprise Knowledge Processing

2025-06-20

AI Summary

This work addresses the challenge of unifying structured database and unstructured text processing in enterprise-grade question-answering systems while ensuring answer accuracy and contextual relevance. We propose a dual-module collaborative architecture: (1) a Text-to-SQL planning module for natural language querying over relational databases, and (2) a hybrid Retrieval-Augmented Generation (RAG) module with citation verification feedback loop, integrating dense/sparse retrieval, commercial re-rankers, and retrieval-generation co-optimization. Our key contribution is the novel citation verification闭环 mechanism, enabling answer traceability, strong grounding, and high-fidelity generation. Evaluated on the RAGTruth benchmark, our approach significantly outperforms the FAISS baseline, achieving marked improvements in contextual relevance and generation quality. Moreover, it supports fine-grained grounding control—critical for high-stakes domains such as finance and healthcare.

Technology Category

• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Knowledge Representation and Reasoning: Argumentation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Community question answering

Abstract

We introduce eSapiens, a unified question-answering system designed for enterprise settings, which bridges structured databases and unstructured textual corpora via a dual-module architecture. The system combines a Text-to-SQL planner with a hybrid Retrieval-Augmented Generation (RAG) pipeline, enabling natural language access to both relational data and free-form documents. To enhance answer faithfulness, the RAG module integrates dense and sparse retrieval, commercial reranking, and a citation verification loop that ensures grounding consistency. We evaluate eSapiens on the RAGTruth benchmark across five leading large language models (LLMs), analyzing performance across key dimensions such as completeness, hallucination, and context utilization. Results demonstrate that eSapiens outperforms a FAISS baseline in contextual relevance and generation quality, with optional strict-grounding controls for high-stakes scenarios. This work provides a deployable framework for robust, citation-aware question answering in real-world enterprise applications.

Problem

• Bridging structured databases and unstructured text for enterprise QA
• Enhancing answer faithfulness with hybrid RAG and verification
• Improving contextual relevance and reducing hallucinations in LLMs

Innovation

• Dual-module architecture for unified QA
• Hybrid RAG pipeline with dense-sparse retrieval
• Citation verification loop ensures answer faithfulness