MortarBench: Evaluating Mortgage Loan Origination Agents

2026-06-17

AI Summary

This work addresses the absence of publicly available evaluation benchmarks for mortgage lending intelligent agents in realistic, complex scenarios. We propose MortarBench, the first comprehensive benchmark specifically designed for mortgage origination, which leverages a financial data synthesis and perturbation pipeline to generate loan approval tasks that reflect real-world distributions and encompass a wide range of edge cases. Using this benchmark, we systematically evaluate large language models on accuracy, risk control, and fairness. Our analysis reveals that leading closed-source models exhibit systematic bias against non-English names, achieving a maximum exact-match accuracy of only 77.1%. To mitigate this, we introduce the CRIT confidence calibration framework, which improves accuracy to 80.5% while significantly reducing bias and enhancing risk management capabilities.

Abstract

Loan origination is the process by which a lender creates a new loan, from application and underwriting through approval and funding. This process serves a critical role in evaluating the eligibility and level of risk posed by an applicant. Recently, firms have begun using mortgage loan agents to augment human loan officers, despite a lack of any public benchmark. To fill this gap, we present MortarBench, a loan origination agent benchmark. MortarBench uses a financial data synthesis and mutation pipeline to generate examples with broad edge case coverage that match real-world distributions and questions. We find that state-of-the-art large language models (LLMs) perform poorly, with closed-source models achieving at most 77.1\% exact match accuracy. We also discover systematic biases in LLM perception of foreignness related to non-English names. Noting these weaknesses, we introduce CRIT, a confidence calibration framework. Our method increases accuracy to 80.5\% while improving risk management steering and reducing bias.

Problem

• mortgage loan origination
• agent benchmark
• large language models
• bias evaluation
• financial decision-making

Innovation

• MortarBench
• loan origination agents
• financial data synthesis
• confidence calibration
• LLM bias mitigation