Curated retrieval versus open web search in public AI information services: a coverage-trust trade-off

2026-07-06

AI Summary

This study addresses the inherent trade-off between coverage breadth and information reliability in public AI information services by empirically comparing retrieval-augmented generation (RAG) systems built on curated corpora against open web search in the context of EU governmental question-answering. Through expert evaluation, it identifies “source credibility” as a latent yet quantifiable dimension of response quality. Findings reveal that while open web search achieves broader coverage, 35% of its cited sources are either untrustworthy or irrelevant; conversely, curated corpora ensure high source reliability but suffer from limited scope. Moreover, system prompts exhibit minimal efficacy in steering models toward citing trustworthy domains. These results provide empirical grounding and practical guidance for designing information-source strategies in public-facing AI systems.

Abstract

Public institutions increasingly use large language models (LLMs) to answer citizens' questions, often pairing a curated knowledge base with live web search, yet whether the sources behind these answers can be trusted has received little empirical scrutiny. We report a pre-launch expert evaluation of Evrópuvefur, an independent, government-funded service run by the University of Iceland that answers questions about the European Union, conducted as Iceland prepared for its referendum of 29 August 2026 on whether to resume EU accession talks. Five domain experts produced 551 evaluations of 449 AI-generated answers, scoring each against a seven-criterion quality rubric and, separately, flagging individual cited sources. We compared two retrieval paths: a curated local corpus (RAG) and open web search. In more than a third of the reviewed web-search answers (35%, 65 of 187), at least one cited source was flagged, almost always as untrustworthy or irrelevant; curated sources were flagged far less often and only for being out of date. Web search answered more questions, but at the cost of source quality; the curated corpus was trustworthy yet limited in coverage, and the model declined to respond when it fell short. The citation mix also passed over strong sources: across all 287 web-search answers, the system never cited RÚV, the public broadcaster and the country's most widely used news source. A companion prompt ablation shows how weak prompt-level steering is: a trusted-domain list in the system prompt raised the share of citations to listed domains only from 12% to 21%. Fluency and topical fit did not predict source trustworthiness. We argue that source trustworthiness is a measurable yet largely invisible dimension of information quality in public AI services, and we discuss transparency-oriented responses and their trade-offs.

Problem

• curated retrieval
• open web search
• source trustworthiness
• coverage-trust trade-off
• public AI services

Innovation

• curated retrieval
• open web search
• source trustworthiness
• retrieval-augmented generation
• public AI services