CIVI: A Framework for Diagnosing Search Agent Failures in Civic Information

2026-09-08

AI Summary

为解决公共部门中大型语言模型提供错误信息的问题，提出CIVI框架，通过ARISE方法分析搜索代理失败原因，发现72.1%的失败归因于检索限制。

Abstract

Large Language Models are increasingly deployed in public-sector settings, where incorrect guidance can cause irreversible harm. We introduce CIVI, the first framework for diagnosing search agent failures in civic information. Its benchmark instantiation jointly spans cross-national, interjurisdictional government contexts (federal, state, and local) and functional categories from an internationally adopted United Nations standard. We evaluate ten frontier search agents and find that none matches an attentive human baseline. Alongside accuracy, CIVI measures search invocation rate, selective no-search accuracy, and how often agents cite authoritative government sources. To perform this diagnosis, we introduce ARISE, which decomposes agentic search failures into four mutually exclusive modes, isolated via source-injection ablation. ARISE attributes 72.1% of all observed failures to retrieval-bound causes rather than to gaps in the models'parametric knowledge.

Problem

• Large Language Models
• search agent failures
• civic information
• public-sector settings

Innovation

• CIVI
• public-sector information
• search agent failures
• ARISE
• retrieval-bound causes