Multi-LLM Collaborative MRI Report Generation for Visual Instruction Tuning in Brain Oncology

2026-07-16

AI Summary

Existing medical vision-language models struggle to handle 3D imaging, and the scarcity of paired 3D brain MRI and textual report data has hindered their application in neuro-oncology. To address this, this work proposes a collaborative multi-large language model (LLM) framework that automatically generates and verifies diagnostic reports based on 3D brain MRI scans. The study introduces the first 3D MRI–text paired dataset specifically curated for brain tumors and successfully extends vision instruction tuning to the 3D medical imaging domain. The resulting specialized vision-language model significantly outperforms current 2D and 3D approaches in both MRI report generation and visual question answering tasks, markedly improving report quality and advancing precision diagnosis and treatment of brain tumors.

Abstract

Recent advances in large language models (LLMs) and their extension to vision-language models (VLMs) have made it easier to combine text and images for tasks such as report generation. Existing VLMs in medicine typically focus on 2D images (chest X-rays), and their extension to 3D imaging has been difficult because of the lack of paired 3D imaging-text data. Thus, we introduce a new method for creating a 3D image-text dataset for brain oncology using 3D MRI scans of glioma and meningioma cases. We use a cooperative system in which several LLMs work together to generate and check reports, ensuring that they are accurate and clear. By leveraging the new 3D MRI-text dataset, we further build a VLM that converts MRI scans into tokens and aligns them with text instructions. Our VLM performed better in report generation and visual question answering tasks than other 2D and 3D methods. Our method not only improves the quality of reports but also helps with better diagnosis and treatment in brain oncology.

Problem

• 3D MRI
• medical report generation
• vision-language models
• brain oncology
• image-text dataset

Innovation

• Multi-LLM collaboration
• 3D MRI-text dataset
• visual instruction tuning
• brain oncology
• vision-language model