Efficient Multimodal Clinical Question Answering for Pulmonary Embolism Risk Assessment

2026-06-21

AI Summary

This study addresses the challenges of multimodal data integration and efficient risk assessment in the clinical diagnosis and management of pulmonary embolism (PE). We develop a clinical question-answering system based on compact multimodal large language models (e.g., Gemma2-2B, Gemma2-4B), leveraging the INSPECT dataset to support inputs from CT pulmonary angiography (CTPA) images, electronic health records (EHR), and their fusion, using zero-shot and few-shot prompting strategies. To our knowledge, this is the first systematic evaluation of such models for diagnostic and prognostic tasks in PE. Results demonstrate that the model integrating CTPA and EHR achieves the best performance, with diagnostic accuracy significantly surpassing that of prognostic tasks such as readmission prediction, thereby highlighting the potential of compact multimodal models for early PE risk identification and interpretability.

Abstract

Pulmonary embolism (PE) is a high risk cardiopulmonary condition whose management requires both timely diagnosis and reliable assessment of future clinical risk. Because PE care routinely combines computed tomography pulmonary angiography (CTPA), radiology interpretation, and longitudinal electronic health record (EHR) evidence, it provides a clinically meaningful setting for evaluating compact multimodal language models. In this work, we build a benchmark using efficient multimodal large language models (MLLMs) on INSPECT, a multimodal PE dataset containing 23,248 CTPA studies from 19,402 patients. We formulate eight diagnostic and prognostic tasks as structured clinical question answering problems and evaluate on typical efficient MLLMs under CTPA-Only, EHR-Only, and CTPA+EHR settings with zero-shot and few-shot prompting. Results show that Gemma4 E4B and Gemma4 E2B perform more strongly when EHR evidence is available, especially under CTPA+EHR input. Task level analysis further shows that PE diagnosis achieves higher performance than prognostic tasks, particularly readmission prediction. These observations suggest that compact multimodal models have the great potential in early stage PE risk detection and explanation.

Problem

• Pulmonary Embolism
• Multimodal Clinical Question Answering
• Risk Assessment
• CTPA
• Electronic Health Record

Innovation

• multimodal large language models
• pulmonary embolism
• clinical question answering
• electronic health records
• CTPA