🤖 AI Summary
Heparin dosing personalization faces critical bottlenecks: heavy reliance on scarce real-patient data and difficulty in designing clinically meaningful reward functions for reinforcement learning. Method: This paper proposes an offline model-guided inverse reinforcement learning (IRL) framework that implicitly infers the reward function from limited clinical expert decision data. It integrates a parameterized reward network with physiologically grounded pharmacokinetic–pharmacodynamic (PK-PD) priors, eliminating the need for handcrafted explicit rewards and avoiding unsafe online trial-and-error. Contribution/Results: We introduce the novel “offline model-guided” paradigm, enhancing both interpretability and cross-drug generalizability. In heparin dose recommendation, the learned policy significantly improves activated partial thromboplastin time (aPTT) target attainment. Validation via both inferred reward structure and alignment with clinical biomarkers confirms accurate recovery of expert therapeutic intent.
📝 Abstract
Accurate medication dosing holds an important position in the overall patient therapeutic process. Therefore, much research has been conducted to develop optimal administration strategy based on Reinforcement learning (RL). However, Relying solely on a few explicitly defined reward functions makes it difficult to learn a treatment strategy that encompasses the diverse characteristics of various patients. Moreover, the multitude of drugs utilized in clinical practice makes it infeasible to construct a dedicated reward function for each medication. Here, we tried to develop a reward network that captures clinicians' therapeutic intentions, departing from explicit rewards, and to derive an optimal heparin dosing policy. In this study, we introduce Offline Model-based Guided Reward Learning (OMG-RL), which performs offline inverse RL (IRL). Through OMG-RL, we learn a parameterized reward function that captures the expert's intentions from limited data, thereby enhancing the agent's policy. We validate the proposed approach on the heparin dosing task. We show that OMG-RL policy is positively reinforced not only in terms of the learned reward network but also in activated partial thromboplastin time (aPTT), a key indicator for monitoring the effects of heparin. This means that the OMG-RL policy adequately reflects clinician's intentions. This approach can be widely utilized not only for the heparin dosing problem but also for RL-based medication dosing tasks in general.