Risk Is Not Review Value: Wrong-Answer Exposure Under Bounded Review Budgets

2026-09-07

AI Summary

论文针对有限审核预算下LLM生成答案的评估问题，提出了一种基于错误答案暴露减少的优先级排序方法，并通过WAER和PRRE指标验证了其有效性。

Abstract

LLM assistants often produce more answers than humans can review before users see them. Most evaluations ask whether an answer is wrong, unsupported, or low-confidence. Bounded review budgets instead ask which answers should be checked first under a fixed review budget. Risk alone is not enough: a high-risk answer may be hard to repair, while a moderately risky answer may be directly correctable from available evidence. For generated-answer evaluation, we model review prioritization as exposure reduction, where review value combines estimated wrongness, intervention affordance, impact, and cost. We evaluate review queues with Wrong-Answer Exposure Ratio (WAER), the fraction of wrong answers left unreviewed, and post-repair residual exposure (PRRE), the fraction still exposed after deterministic benchmark-supported repairs. PRRE uses repairability rules that do not numerically reuse the affordance scores used for ranking. On a 720-item TAT-QA/SciFact stress benchmark, review-value ranking keeps answer-level WAER nearly unchanged at 20% budget (0.605 vs. 0.600) but lowers PRRE from 0.881 to 0.716. These results show that trustworthy LLM evaluation should measure not only error detection, but also how limited review capacity reduces exposed wrong answers.

Problem

• Bounded Review Budgets
• Answer Evaluation
• Exposure Reduction
• Wrong-Answer Exposure Ratio
• Post-Repair Residual Exposure

Innovation

• exposure reduction
• review value
• limited review budget
• PRRE