MSM-Mem: A Universal Medical Structured Multimodal Memory Framework for Medical AI Agents

2026-08-22

AI Summary

为解决医疗AI代理缺乏经验积累的问题，提出MSM-Mem框架，通过组织和更新多模态记忆以提升决策能力。

Abstract

Clinical decision-making is inherently experience-driven: physicians progressively refine their reasoning by synthesizing patient history, multimodal observations, and prior diagnostic experiences across interactions. In contrast, current multimodal large language model (MLLM)-based medical AI agents largely operate as stateless inference systems, generating decisions independently for each interaction without retaining or internalizing experiential knowledge. This discrepancy limits their ability to progressively improve reasoning reliability through usage and adapt to longitudinal patient contexts in real-world clinical workflows. In this study, we propose Medical Structured Multimodal Memory (MSM-Mem), an agentic memory framework that enables medical AI agents to evolve through accumulated clinical experiences. MSM-Mem organizes heterogeneous clinical experiences into semantic, episodic, and visual memory and incrementally updates them during inference, allowing the agent to retrieve prior experiences to inform current reasoning and progressively refine decision-making over time. Evaluations on MoE-LLaVA backbones demonstrate consistent performance improve- ments with further gains observed through continued usage. In general, MSM-Mem offers a viable pathway toward medical AI agents capable of evolving their reasoning competence in a manner analogous to the way clinicians learn from practice over time.

Problem

• multimodal large language model
• medical AI agents
• clinical decision-making
• experiential knowledge
• longitudinal patient contexts

Innovation

• Medical Structured Multimodal Memory
• clinical experiences
• incremental updates
• reasoning improvement
• evolving AI agents