Natural Language-Assisted Multi-modal Medication Recommendation

2024-10-21

International Conference on Information and Knowledge Management

AI Summary

Addressing the challenge of combinatorial drug recommendation for complex chronic diseases, this paper proposes a patient–drug dual-perspective cross-modal alignment framework. It jointly models drug chemical structures—encoded via graph-enhanced representations—and functional textual descriptions, while synergistically integrating diagnostic, symptom, and procedural text from patients’ electronic health records (EHRs). To overcome limitations of unimodal or shallow fusion approaches, the framework employs a domain-adapted pretrained language model coupled with a cross-modal alignment optimization strategy. Evaluated on three public benchmark datasets, our method achieves an average 4.72% improvement in Jaccard similarity over prior state-of-the-art methods. The implementation is publicly available.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Psychology-informed user models and recommender systems
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Combinatorial medication recommendation(CMR) is a fundamental task of healthcare, which offers opportunities for clinical physicians to provide more precise prescriptions for patients with intricate health conditions, particularly in the scenarios of long-term medical care. Previous research efforts have sought to extract meaningful information from electronic health records (EHRs) to facilitate combinatorial medication recommendations. Existing learning-based approaches further consider the chemical structures of medications, but ignore the textual medication descriptions in which the functionalities are clearly described. Furthermore, the textual knowledge derived from the EHRs of patients remains largely underutilized. To address these issues, we introduce the Natural Language-Assisted Multi-modal Medication Recommendation(NLA-MMR), a multi-modal alignment framework designed to learn knowledge from the patient view and medication view jointly. Specifically, NLA-MMR formulates CMR as an alignment problem from patient and medication modalities. In this vein, we employ pretrained language models(PLMs) to extract in-domain knowledge regarding patients and medications, serving as the foundational representation for both modalities. In the medication modality, we exploit both chemical structures and textual descriptions to create medication representations. In the patient modality, we generate the patient representations based on textual descriptions of diagnosis, procedure, and symptom. Extensive experiments conducted on three publicly accessible datasets demonstrate that NLA-MMR achieves new state-of-the-art performance, with a notable average improvement of 4.72% in Jaccard score. Our source code is publicly available on https://github.com/jtan1102/NLA-MMR_CIKM_2024.

Problem

• Natural Language Processing
• Multimodal Information
• Drug Recommendation

Innovation

• NLA-MMR
• Pre-trained Language Model
• Chemical Structure Analysis