Planner-Auditor Twin: Agentic Discharge Planning with FHIR-Based LLM Planning, Guideline Recall, Optional Caching and Self-Improvement

2026-01-28

AI Summary

This work addresses safety concerns in clinical discharge planning with large language models, which are often compromised by hallucinations, omissions, and poor confidence calibration. To mitigate these issues, the authors propose a Planner–Auditor dual-module framework: the Planner generates structured discharge plans from FHIR data while estimating its own confidence, and the Auditor validates these outputs through deterministic multitask coverage assessment and calibration monitoring. A discrepancy-buffered replay mechanism is introduced to enable self-improvement by prioritizing high-confidence, low-coverage samples. Without retraining the underlying model, this approach increases task coverage from 32% to 86%, substantially improves confidence calibration, reduces high-confidence omissions, and thereby enhances the reliability and safety of clinical discharge planning.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Objective: Large language models (LLMs) show promise for clinical discharge planning, but their use is constrained by hallucination, omissions, and miscalibrated confidence. We introduce a self-improving, cache-optional Planner-Auditor framework that improves safety and reliability by decoupling generation from deterministic validation and targeted replay. Materials and Methods: We implemented an agentic, retrospective, FHIR-native evaluation pipeline using MIMIC-IV-on-FHIR. For each patient, the Planner (LLM) generates a structured discharge action plan with an explicit confidence estimate. The Auditor is a deterministic module that evaluates multi-task coverage, tracks calibration (Brier score, ECE proxies), and monitors action-distribution drift. The framework supports two-tier self-improvement: (i) within-episode regeneration when enabled, and (ii) cross-episode discrepancy buffering with replay for high-confidence, low-coverage cases. Results: While context caching improved performance over baseline, the self-improvement loop was the primary driver of gains, increasing task coverage from 32% to 86%. Calibration improved substantially, with reduced Brier/ECE and fewer high-confidence misses. Discrepancy buffering further corrected persistent high-confidence omissions during replay. Discussion: Feedback-driven regeneration and targeted replay act as effective control mechanisms to reduce omissions and improve confidence reliability in structured clinical planning. Separating an LLM Planner from a rule-based, observational Auditor enables systematic reliability measurement and safer iteration without model retraining. Conclusion: The Planner-Auditor framework offers a practical pathway toward safer automated discharge planning using interoperable FHIR data access and deterministic auditing, supported by reproducible ablations and reliability-focused evaluation.

Problem

• hallucination
• omission
• miscalibrated confidence
• discharge planning
• clinical safety

Innovation

• Planner-Auditor framework
• FHIR-based LLM
• self-improvement
• calibration
• discrepancy buffering