World Feedback for Clinical Agents: Diagnosing RL in FHIR Environments

2026-07-01

AI Summary

This work addresses the challenge that reinforcement learning (RL) agents in clinical FHIR environments often exhibit passive or inactive behavior due to sparse feedback and insufficient foundational capabilities. To diagnose the structural limitations of RL in clinical decision-making tasks, the authors introduce MedAgentBench-v3, a benchmark featuring a world-feedback validator that operates without requiring per-turn annotations. They further propose a tripartite taxonomy—encompassing decision logic, format knowledge, and query formulation—to uncover the fundamental non-learnable barriers underlying RL failure. Building on the Qwen3-8B model, they advocate a hybrid training paradigm combining supervised fine-tuning (SFT) to inject code-generation capabilities with RL to acquire conditional reasoning. Experiments show that pure RL achieves only 18.2% pass@1, whereas rule-guided SFT improves performance to 34.1%, with the gap attributed to identified deficits in format knowledge and actionable decision competence.

Abstract

Clinical protocol-execution tasks -- checking a lab value, applying a threshold, placing a correctly structured FHIR order -- are natural candidates for RL from world feedback: once clinical SMEs encode decision logic into a verifier, that verifier grades unlimited rollouts without per-episode annotation. But applying RL requires a sound feedback channel and sufficient base capability. We audit MedAgentBench v1/v2, find a 41.7\% silent-finish ceiling that makes inaction the RL dominant strategy, and construct \textbf{MedAgentBench-v3 (MAB-v3)} (508 tasks, 8.9\% ceiling). Training Qwen3-8B exposes two structural barriers: a \emph{capability ceiling} (10/20 task types have 0\% base performance, zero gradient) and a \emph{format-knowledge barrier} (3/20 types require exact clinical codes undiscoverable by exploration). Pure RL reaches 18.2\% pass@1 vs.\ 34.1\% for rule-based SFT; the 15.9~pp gap is attributable entirely to these barriers. A decision/format-knowledge/lookup taxonomy predicts RL learnability and prescribes the fix: SFT to inject codes, RL to learn conditionals.

Problem

• Reinforcement Learning
• Clinical Agents
• FHIR
• Capability Ceiling
• Format-Knowledge Barrier

Innovation

• world feedback
• reinforcement learning
• FHIR
• MedAgentBench
• supervised fine-tuning