ConRub-Med: Reinforcement Learning with Consensus Rubrics for Open-Ended Medical Question Answering

2026-08-11

AI Summary

This work addresses the challenge of evaluating open-ended medical question answering, where existing approaches lack verifiable, fine-grained reward mechanisms to assess partial correctness, missing information, or clinically critical errors. To this end, the authors propose a method for generating atomic scoring criteria based on multi-model consensus, integrating a three-state evaluation scheme—correct coverage, missing information, and erroneous statements—with an enhanced Group Relative Policy Optimization (GRPO) framework that incorporates unbiased ordinal advantage estimation to improve clinical relevance. The proposed approach achieves state-of-the-art performance, ranking first on six out of nine benchmarks and attaining a HealthBench-Hard score of 38.98 ± 1.04, significantly outperforming current methods. Expert blind evaluations further confirm its superior clinical utility.

Abstract

Reinforcement learning with verifiable rewards has been especially effective in mathematics and coding, where answers can be checked automatically. Many open-ended medical questions lack comparably cheap outcome verifiers: responses may be partly correct, incomplete, or contain clinically consequential errors. Rubrics written or validated by physicians offer strong clinical grounding, but involving experts in every instance is costly. Model-generated rubrics make this supervision scalable. We introduce ConRub-Med to preserve useful distinctions as rubric feedback moves from construction to policy optimization. For each prompt, three heterogeneous language models propose atomic criteria independently; a separate model reviews them, retaining only criteria with semantic support from all three generators. Three-State scoring distinguishes correct coverage, missing information, and incorrect claims. Errors receive negative rather than zero credit. When every response in a complete Group Relative Policy Optimization (GRPO) group receives the same final reward, a pairwise judge provides sequence advantages only if both candidate orders agree, without changing the scalar rewards. Groups without ties use vanilla GRPO. In a blinded study matched by question, two medical experts rate panels from the full pipeline as more clinically relevant than panels produced by one generator. Across the evaluated open models, ConRub-Med ranks first on six of nine benchmarks and achieves the highest medical and generalization averages. Using the resulting rubric dataset of 5,166 prompts, it scores $38.98 \pm 1.04$ (mean $\pm$ SD) on HealthBench-Hard, compared with InfiMed-ORBIT's 33.60 with 8,000 samples and 37.30 with 28,000.

Problem

• open-ended medical question answering
• reinforcement learning
• clinical correctness
• reward verification
• rubric-based evaluation

Innovation

• Consensus Rubrics
• Three-State Scoring
• Group Relative Policy Optimization
• Reinforcement Learning
• Medical Question Answering