FinRAG-12B: A Production-Validated Recipe for Grounded Question Answering in Banking

2026-05-06

AI Summary

This study addresses the challenges of accuracy, regulatory compliance, and verifiability in deploying large language models (LLMs) within the banking sector by proposing a data-efficient, end-to-end framework. The framework integrates LLM-as-a-Judge filtering, citation annotation, curriculum learning, and a calibrated rejection mechanism, while supporting quantized deployment across the entire pipeline—from data construction to efficient inference. A domain-specific model trained on only 143 million tokens surpasses GPT-4.1 in citation accuracy and demonstrates substantially improved rejection behavior on unanswerable queries. Deployed across more than 40 financial institutions, the system increases query resolution rates by 7.1 percentage points (p<0.001), accelerates response times by 3–5×, and reduces inference costs by 20–50×.

Abstract

Large language models (LLMs) are rapidly being adopted across various domains. However, their adoption in banking industry faces resistance due to demands for high accuracy, regulatory compliance, and the need for verifiable and grounded responses. We present a unified, data-efficient framework for training grounded domain-specific LLMs that optimizes answer quality, citation grounding, and calibrated refusal under real-world deployment constraints. First, we describe a data generation pipeline that combines LLM-as-a-Judge filtering, citation annotation, and curriculum learning with only 143M tokens. The resulting 12B model achieves high answer quality outperforming GPT-4.1 on citation grounding, with a modest citation tradeoff versus the untuned base. Second, we propose a calibrated refusal mechanism: training on 22% unanswerable examples yield a 12% "I don't know" rate, substantially improving over the base model's unsafe 4.3% rate while avoiding GPT-4.1's over-refusal (20.2%). Third, we present an end-to-end methodology spanning from data curation to quantized serving. The system is deployed at 40+ financial institutions, achieving a 7.1 percentage point improvement in query resolution (p < 0.001). Additionally, the model delivers 3-5x faster responses at 20-50x lower cost compared to GPT-4.1.

Problem

• grounded question answering
• banking
• regulatory compliance
• verifiable responses
• large language models

Innovation

• grounded question answering
• calibrated refusal
• data-efficient training
• domain-specific LLM
• production deployment