RECOM: A Validity Discrimination Tradeoff in Automatic Metrics for Open Ended Reddit Question Answering

2026-06-17

AI Summary

This study addresses the challenge that existing automatic evaluation metrics for open-ended question answering struggle to simultaneously ensure content validity—distinguishing genuine responses from random noise—and discriminative power—differentiating between stronger and weaker models. To investigate this trade-off, the authors construct the RECOM dataset, comprising 15,000 questions from r/AskReddit along with authentic community responses, and introduce a randomized misalignment baseline as a noise reference. They systematically evaluate metrics including cosine similarity, BERTScore, and LLM-based judges, revealing for the first time a fundamental tension between validity and discriminability: cosine similarity exhibits strong validity but poor discrimination; BERTScore’s discriminative capability sharply declines when response length is controlled; and even LLM judges show limited performance. The work advocates reporting both dimensions alongside a random baseline for comprehensive generation quality assessment.

Abstract

Automatic metrics are the default for evaluating LLM-generated text, yet a metric is quietly asked to do two jobs: tell genuine content alignment from surface coincidence (validity), and tell a better system from a worse one (discriminative power). On open-ended, opinion-driven question answering, the two are in tension. We introduce RECOM (Reddit Evaluation for Correspondence of Models), a contamination-free evaluation dataset of 15,000 r/AskReddit questions (September 2025), each paired with its authentic community replies, which postdate every evaluated model's training cutoff. Scoring five open-source LLMs (7--10B) against every reply each metric paired with a random-derangement noise floor we find that no metric does both jobs well. Cosine similarity separates real from random answers (Cohen's $d \approx 2$) but cannot rank the five models ($|d| < 0.1$); BERTScore precision appears to rank the models (raw $|d|$ up to 0.63), but once response length is controlled this collapses to $|d| = 0.09$ and its validity is weak ($d \approx 0.8$, versus cosine's $\approx 2$). Because every metric scores the same outputs, this validity--discrimination tradeoff is a property of the metrics, not the models, and we argue it stems from representation design. Three independent LLM judges reproduce the validity gap and likewise separate the five models only weakly. We recommend reporting metrics on both axes, with an explicit random-baseline floor. RECOM is publicly available at https://anonymous.4open.science/r/recom-D4B0

Problem

• automatic metrics
• validity
• discriminative power
• open-ended question answering
• evaluation tradeoff

Innovation

• validity-discrimination tradeoff
• automatic evaluation metrics
• RECOM dataset
• open-ended question answering
• LLM evaluation