ProtoMedAgent: Multimodal Clinical Interpretability via Privacy-Aware Agentic Workflows

2026-05-13

AI Summary

This work addresses the limitations of existing interpretable prototype networks in clinical report generation, which often lack structured semantics, and the tendency of retrieval-augmented generation (RAG) approaches to suffer from “retrieval sycophancy” and generate spurious explanations. The authors propose the first clinical reporting framework integrating neuro-symbolic constraints, semantic privacy gating, and a Scribe-Critic reflection mechanism. Their method distills multimodal features into discrete semantic memory atop a frozen prototype backbone, enforces narrative reliability through set-theoretic difference constraints and zero-gradient test-time optimization, and incorporates k-anonymity and ℓ-diversity for privacy preservation. Evaluated on a cohort of 4,160 patients, the approach achieves 91.2% comparative set fidelity—substantially outperforming standard RAG at 46.2%—and reduces membership inference risk by an absolute 9.8%.

Abstract

While interpretable prototype networks offer compelling case-based reasoning for clinical diagnostics, their raw continuous outputs lack the semantic structure required for medical documentation. Bridging this gap via standard Retrieval-Augmented Generation (RAG) routinely triggers ``retrieval sycophancy,'' where Large Language Models (LLMs) hallucinate post-hoc rationalizations to align with visual predictions. We introduce ProtoMedAgent, a framework that formalizes multimodal clinical reporting as an iterative, zero-gradient test-time optimization problem over a strict neuro-symbolic bottleneck. Operating on a frozen prototype backbone, we distill latent visual and tabular features into a discrete semantic memory. Online generation is strictly constrained by exact set-theoretic differentials and a reflective Scribe-Critic loop, mathematically precluding unsupported narrative claims. To safely bound data disclosure, we introduce a semantic privacy gate governed by $k$-anonymity and $\ell$-diversity. Evaluated on a 4,160-patient clinical cohort, ProtoMedAgent achieves 91.2\% Comparison Set Faithfulness where it fundamentally outperforms standard RAG (46.2\%). ProtoMedAgent additionally leverages a binding $\ell$-diversity phase transition to systematically reduce artifact-level membership inference risks by an absolute 9.8\%.

Problem

• clinical interpretability
• retrieval sycophancy
• multimodal reporting
• semantic structure
• hallucination

Innovation

• ProtoMedAgent
• neuro-symbolic bottleneck
• retrieval sycophancy
• semantic privacy gate
• zero-gradient test-time optimization