How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

2026-09-01

AI Summary

本文针对开放式问答评估难题，提出了一种语义正确性框架，包括分类体系、基准数据集及基于上下文的精度指标CAP，以更好地区分答案质量。

Abstract

Reliable evaluation of open-ended question answering remains a bottleneck for measuring answer correctness of modern LLMs. Unlike multiple-choice tasks, free-form answers may be correct in many surface forms and may fail in qualitatively different ways, including incompleteness, contradiction, overgeneration, and endorsement of false premises. Existing judgment-based and similarity-based metrics often collapse these distinctions. We address this gap with three reusable contributions. First, we introduce a semantic correctness taxonomy that assigns open-ended answers to eight ordered classes, separating verbose-but-correct answers from those contaminated by hallucinated content. Second, we release CAP-Correctness, an 8.8k-example benchmark spanning widely used QA datasets, and CAP-Statements, an 11k-example dataset for converting question-answer pairs into declarative statements for natural language inference (NLI) training and statement-based evaluation. Third, we introduce CAP (Context-Aware Precision), a reference-based metric that scores question-conditioned statements using bidirectional NLI. Under a monotonicity protocol testing whether metrics respect the taxonomy's intended ordering, CAP outperforms established baselines.

Problem

• open QA
• answer correctness
• semantic correctness
• evaluation metrics
• large language models

Innovation

• Semantic Correctness Taxonomy
• CAP-Correctness
• CAP-Statements
• Context-Aware Precision (CAP)
• Bidirectional NLI