Scalably Enhancing the Clinical Validity of a Task Benchmark with Physician Oversight

2025-12-22

AI Summary

Clinical risk-scoring benchmarks such as MedCalc-Bench suffer from the entrenchment of historically erroneous models as de facto “gold standards,” particularly problematic when used as reward signals in reinforcement learning, where such biases are amplified. Method: We propose a “living documentation” benchmark paradigm featuring physician-in-the-loop, low-burden dynamic curation: GRPO-based RL fine-tuning of Qwen3-8B, multi-stage agentic verification, LLM-driven logical consistency checking, and clinical-knowledge-guided controversy identification—enabling scalable, trustworthy re-annotation. Contribution/Results: Experiments uncover substantial label noise; post-correction, model accuracy improves by 8.7 percentage points. This work is the first systematic demonstration that safety-critical domain benchmarks require continuous governance, establishing a novel human–AI collaborative paradigm for dynamic benchmark auditing.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Automating the calculation of clinical risk scores offers a significant opportunity to reduce physician administrative burden and enhance patient care. The current standard for evaluating this capability is MedCalc-Bench, a large-scale dataset constructed using LLM-based feature extraction and rule-based aggregation. However, treating such model-generated benchmarks as static oracles risks enshrining historical model errors as evaluation gold standards, a problem dangerously amplified when these datasets serve as reward signals for Reinforcement Learning (RL). In this work, we propose viewing benchmarks for complex tasks such as clinical score computation as ''in-progress living documents'' that should be periodically re-evaluated as the processes for creating them improve. We introduce a systematic, physician-in-the-loop pipeline that leverages advanced agentic verifiers to audit and relabel MedCalc-Bench, utilizing automated triage to reserve scarce clinician attention for the most contentious instances. Our audit reveals that a notable fraction of original labels diverge from medical ground truth due to extraction errors, calculator logic mismatches, and clinical ambiguity. To study whether this label noise meaningfully impacts downstream RL training, we fine-tune a Qwen3-8B model via Group Relative Policy Optimization (GRPO) and demonstrate that training on corrected labels yields an 8.7% absolute improvement in accuracy over the original baseline -- validating that label noise materially affects model evaluation. These findings underscore that in safety-critical domains, rigorous benchmark maintenance is a prerequisite for genuine model alignment.

Problem

• Enhances clinical benchmark validity via physician oversight
• Addresses label noise from model-generated benchmark errors
• Improves model accuracy by correcting benchmark labels

Innovation

• Physician-in-the-loop pipeline for benchmark auditing
• Agentic verifiers with automated triage for contentious cases
• Corrected labels improve RL training accuracy by 8.7%