RASC+: Retrieval-Constrained LLM Adjudication for Clinical Value Set Authoring

2026-06-22

AI Summary

This work addresses the limitations of large language models in clinical value set construction, where their inability to reliably memorize extensive, version-controlled standardized terminologies hinders performance. The authors propose a retrieval-constrained two-stage adjudication mechanism: first, a high-recall retrieval pipeline—integrating Qwen3, lexicon-aware query expansion, and fallback retrieval—constructs an auditable candidate code pool; then, GPT-5 performs blind adjudication within this constrained pool, replacing conventional cross-encoder selection. Evaluated on the RASC benchmark, the approach improves macro F1 from 0.287 to 0.549 overall, achieves 0.533 on an independent publisher subset, and attains a candidate pool recall of 0.730, substantially enhancing both completion accuracy and result safety.

Abstract

Clinical value sets define the standardized terminology codes used in quality measurement, phenotyping, cohort construction, and clinical decision support. The recently introduced Retrieval-Augmented Set Completion (RASC) benchmark showed that direct zero-shot large language model (LLM) generation is poorly suited to this task: clinical code systems are large, version-controlled, and not reliably memorized by language models. We study a stage-wise alternative in which candidate-pool construction is optimized for recall and a constrained LLM adjudicator is optimized for candidate selection. On the full 3,744-value-set RASC test split, Qwen3-based retrieval with vocabulary-aware expansion and code-display rescue retrieval increases candidate-pool recall from the original RASC retrieval baseline of 0.553 to 0.730; on the held-out-publisher stratum, pool recall is 0.655. The higher-recall pool alone is not sufficient: applying the original SAPBert cross-encoder to this expanded pool gives full-test macro F1 of 0.287 and held-out-publisher macro F1 of 0.233. Replacing the stage-2 selector with blinded GPT-5 adjudication over the same pool increases full-test macro F1 to 0.549 and held-out-publisher macro F1 to 0.533. These results show that retrieval-constrained LLM adjudication can substantially improve value set completion while preserving the safety constraint that all returned codes must come from an auditable candidate pool.

Problem

• clinical value sets
• terminology codes
• large language models
• code retrieval
• zero-shot generation

Innovation

• Retrieval-Constrained LLM Adjudication
• Clinical Value Set Authoring
• Vocabulary-Aware Retrieval
• Stage-wise Candidate Selection
• Auditable Code Pool