Flexible Blood Glucose Control: Offline Reinforcement Learning from Human Feedback

2025-01-27

AI Summary

Existing reinforcement learning (RL) methods for insulin regulation in type 1 diabetes struggle to incorporate patient-specific preferences, resulting in non-personalized and clinically uninterpretable control policies. To address this, we propose PAINT—a novel framework that synergistically integrates sketch-based reward modeling with safety-constrained offline RL, enabling continuous preference tuning, preprandial glucose anticipation, and device fault tolerance. PAINT incorporates expert-guided preference annotation, safety-aware action constraints, and few-shot robust training. In silico evaluation demonstrates a 15% reduction in glycemic risk, a 10% improvement in postprandial time-in-target range, and a 1.6% decrease in glucose variability under device anomalies. Moreover, the method exhibits strong robustness to annotation noise and low-data regimes. This work represents the first approach achieving simultaneous safety guarantees, adjustable personalization, and clinical interpretability in closed-loop glucose control.

Technology Category

• Knowledge Representation and Reasoning: Preferences
• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Reinforcement Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Security and Privacy: Privacy-enhancing technologies

Abstract

Reinforcement learning (RL) has demonstrated success in automating insulin dosing in simulated type 1 diabetes (T1D) patients but is currently unable to incorporate patient expertise and preference. This work introduces PAINT (Preference Adaptation for INsulin control in T1D), an original RL framework for learning flexible insulin dosing policies from patient records. PAINT employs a sketch-based approach for reward learning, where past data is annotated with a continuous reward signal to reflect patient's desired outcomes. Labelled data trains a reward model, informing the actions of a novel safety-constrained offline RL algorithm, designed to restrict actions to a safe strategy and enable preference tuning via a sliding scale. In-silico evaluation shows PAINT achieves common glucose goals through simple labelling of desired states, reducing glycaemic risk by 15% over a commercial benchmark. Action labelling can also be used to incorporate patient expertise, demonstrating an ability to pre-empt meals (+10% time-in-range post-meal) and address certain device errors (-1.6% variance post-error) with patient guidance. These results hold under realistic conditions, including limited samples, labelling errors, and intra-patient variability. This work illustrates PAINT's potential in real-world T1D management and more broadly any tasks requiring rapid and precise preference learning under safety constraints.

Problem

• Reinforcement Learning
• Type 1 Diabetes Management
• Personalization

Innovation

• PAINT
• Personalized Insulin Delivery
• Adaptive Algorithm