MMIR-TCM: Memory-Integrated Multimodal Inference and Retrieval for TCM Clinical Decision Support

2026-07-02

AI Summary

This study addresses the challenges of high subjectivity and poor reproducibility in traditional Chinese medicine (TCM) tongue diagnosis, as well as the semantic gap between visual and textual modalities and the lack of standardized datasets in multimodal AI for syndrome differentiation and prescription generation. To this end, the authors propose a three-stage diagnostic framework that emulates the clinical reasoning process of TCM experts. The approach integrates a training-free, memory-augmented SAM for tongue image segmentation, a Qwen3-VL multimodal model fine-tuned for TCM tasks, and a retrieval-augmented generation system powered by Qwen3, all built upon the authors’ newly curated large-scale multimodal dataset, MedTCM. Key innovations include the introduction of TDEU, a novel metric for evaluating clinical reasoning, and an end-to-end capability for structured tongue analysis and evidence-based prescription generation. Experiments demonstrate that the proposed framework significantly outperforms state-of-the-art models such as GPT-4o and Gemini 2.5 Flash in clinical accuracy.

Abstract

Traditional Chinese Medicine (TCM) diagnosis, particularly through tongue inspection, faces persistent challenges in subjectivity and reproducibility. The application of multimodal artificial intelligence to TCM clinical tasks, such as syndrome differentiation and prescription generation, is significantly hampered by the semantic gap between visual tongue features and textual reasoning, as well as the lack of large-scale, standardized datasets. To address these challenges, we introduce MMIR-TCM, a novel framework that emulates the diagnostic process of TCM experts by integrating multimodal large language model(MLLM) with memory-augmented segmentation and retrieval-augmented generation (RAG). Employing a three-stage architecture, MMIR-TCM integrates a training-free Memory-SAM module for robust tongue extraction, a fine-tuned Qwen3-VL model for structured tongue diagnosis generation, and a Qwen3-based RAG component for evidence-grounded clinical decision support generation. The framework was developed and validated using MedTCM, a new large-scale multimodal dataset that we introduce specifically for advanced TCM research. To properly evaluate our framework's clinical accuracy, which existing metrics fail to capture, we also developed TDEU, a domain-specific evaluation metric incorporating semantic understanding and diagnostic importance. Our comprehensive experiments demonstrate that MMIR-TCM significantly outperforms leading models, including GPT-4o and Gemini 2.5 Flash.

Problem

• Traditional Chinese Medicine
• tongue diagnosis
• multimodal AI
• semantic gap
• clinical decision support

Innovation

• Memory-Augmented Segmentation
• Retrieval-Augmented Generation (RAG)
• Multimodal Large Language Model (MLLM)
• TCM Clinical Decision Support
• Domain-Specific Evaluation Metric