A Unified Vision-Language Model for PSMA PET/CT Report Generation, Visual Question Answering, and Lesion Segmentation

2026-09-14

AI Summary

本文提出了一种统一的PSMA PET/CT视觉-语言模型，用于报告生成、视觉问答和病灶分割，采用LLaVA架构并通过四阶段训练策略优化性能。

Abstract

Accurate PSMA PET/CT interpretation is central to prostate cancer management, yet existing PET/CT AI models typically address isolated tasks. We propose a unified PSMA PET/CT vision-language model for report generation, visual question answering, and lesion segmentation. The framework adopts an LLaVA-style architecture, comprising a PET/CT vision encoder, an MLP-Mixer projection module, a LoRA-tuned large language model, and a 3D segmentation branch. Training followed a four-stage strategy: vision encoder pretraining, projection-layer alignment, VLM fine-tuning, and final multitask tuning. Language tasks used 5,747 PSMA PET/CT datasets with paired reports, while segmentation used the PSMA subset of AutoPET. The model outperformed PET2REP and a CT-based baseline across standard report-generation metrics, improved performance across VQA question types, and achieved higher Dice and lesion-level overlap F1 than SegAnyPET and nnUNet. These results support the feasibility of a unified framework for structured, interactive, interpretable PSMA PET/CT analysis with voxel-level grounding within a single multitask model architecture.

Problem

• PSMA PET/CT
• report generation
• visual question answering
• lesion segmentation
• unified model

Innovation

• Unified Vision-Language Model
• LLaVA-style Architecture
• Multitask Tuning
• PSMA PET/CT
• Voxel-level Grounding