A Human-Centric Evaluation of a Retrieval-Augmented Generation System for Explaining Quebec Insurance Contracts

2026-07-17

AI Summary

This study addresses the “advice gap” faced by Quebec consumers when purchasing auto insurance online, where complex policy language and lack of expert guidance hinder comprehension. To bridge this gap, the authors propose a user-centered explanatory system grounded in Retrieval-Augmented Generation (RAG) and empirically evaluate its effectiveness through a human-centered interaction framework. The findings demonstrate, for the first time from a human-centered perspective, that RAG functions as a “cognitive equalizer,” significantly enhancing contract understandability—particularly benefiting individuals with lower financial literacy. The system excels in user-reported satisfaction, trust, and perceived clarity, with participants especially valuing the sense of autonomy it affords. However, in high-stakes or emotionally charged scenarios, users still exhibit a preference for human assistance over automated support.

Abstract

With the rise of online insurance sales, consumers face a significant \enquote{advice gap}, requiring them to navigate complex legal contracts without expert guidance. This paper presents a human-centric, extrinsic evaluation of a state-of-the-art Retrieval-Augmented Generation system, designed to make Quebec automobile insurance contracts more understandable. Through a user study with 154 participants from Laval University, we assess the agent's real-world utility by measuring system satisfaction, cognitive effort, perceived autonomy, and risk. Our results show the system is perceived as a \enquote{cognitive equalizer}, receiving high ratings for satisfaction, trust, and clarity. Crucially, users value the sense of autonomy the system provides even more than the knowledge itself, with this effect being most pronounced among participants with lower financial literacy, demonstrating how such an agent can directly empower individuals. However, our study also highlights some limitations: participants expressed a strong preference for human agents in high-stakes or emotionally charged scenarios. This work highlights the importance of human-in-the-loop frameworks in ensuring the responsible implementation of AI in high-stakes consumer finance.

Problem

• advice gap
• insurance contracts
• consumer understanding
• financial literacy
• human-AI interaction

Innovation

• Retrieval-Augmented Generation
• human-centric evaluation
• cognitive equalizer
• financial literacy
• human-in-the-loop