Advancing Risk and Quality Assurance: A RAG Chatbot for Improved Regulatory Compliance

2025-07-22

AI Summary

In highly regulated industries, risk and quality (R&Q) compliance faces bottlenecks including complex regulatory interpretation, heavy reliance on domain experts, and low response efficiency. This paper proposes a retrieval-augmented generation (RAG) framework integrating hybrid retrieval (keyword- and semantic-based) with relevance-enhancement mechanisms, coupled with large language models (LLMs), to enable precise and interpretable policy question answering. Key contributions include: (i) a multi-granularity retrieval re-ranking strategy that improves recall accuracy of regulatory clauses; (ii) domain-adapted prompt engineering and context refinement modules to enhance generation reliability; and (iii) systematic hyperparameter analysis for end-to-end performance optimization. Evaluated on 124 expert-annotated real-world queries, our method achieves a 23.6% average F1-score improvement over baseline RAG approaches. The system has been deployed in production, significantly improving frontline staff’s compliance decision-making efficiency and scalability.

Technology Category

• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Risk and Quality (R&Q) assurance in highly regulated industries requires constant navigation of complex regulatory frameworks, with employees handling numerous daily queries demanding accurate policy interpretation. Traditional methods relying on specialized experts create operational bottlenecks and limit scalability. We present a novel Retrieval Augmented Generation (RAG) system leveraging Large Language Models (LLMs), hybrid search and relevance boosting to enhance R&Q query processing. Evaluated on 124 expert-annotated real-world queries, our actively deployed system demonstrates substantial improvements over traditional RAG approaches. Additionally, we perform an extensive hyperparameter analysis to compare and evaluate multiple configuration setups, delivering valuable insights to practitioners.

Problem

• Navigating complex regulatory frameworks in regulated industries
• Overcoming bottlenecks in traditional expert-dependent compliance methods
• Enhancing Risk and Quality query processing with RAG

Innovation

• RAG system with hybrid search
• Leveraging Large Language Models
• Extensive hyperparameter analysis