PET2Rep: Towards Vision-Language Model-Drived Automated Radiology Report Generation for Positron Emission Tomography

2025-08-05

AI Summary

Current vision-language models (VLMs) for medical imaging predominantly focus on structural modalities, largely neglecting PET’s quantitative molecular metabolic features, and lack automated whole-body PET report generation methods and standardized evaluation benchmarks. To address this gap, we introduce PET2Rep—the first large-scale, whole-body PET image–report paired dataset incorporating quantitative metabolic metrics—and establish a fine-grained radiology reporting benchmark covering 30 anatomical regions. We further propose clinically grounded, joint efficiency–accuracy evaluation metrics. Using this framework, we systematically benchmark 30 state-of-the-art general-purpose and medical-domain-specific VLMs. Results reveal substantial deficiencies in current top-performing VLMs’ ability to accurately describe uptake patterns in critical organs, underscoring two fundamental challenges: metabolic-aware representation learning and precise anatomical–functional alignment. These findings highlight urgent research directions for advancing medical VLMs.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Positron emission tomography (PET) is a cornerstone of modern oncologic and neurologic imaging, distinguished by its unique ability to illuminate dynamic metabolic processes that transcend the anatomical focus of traditional imaging technologies. Radiology reports are essential for clinical decision making, yet their manual creation is labor-intensive and time-consuming. Recent advancements of vision-language models (VLMs) have shown strong potential in medical applications, presenting a promising avenue for automating report generation. However, existing applications of VLMs in the medical domain have predominantly focused on structural imaging modalities, while the unique characteristics of molecular PET imaging have largely been overlooked. To bridge the gap, we introduce PET2Rep, a large-scale comprehensive benchmark for evaluation of general and medical VLMs for radiology report generation for PET images. PET2Rep stands out as the first dedicated dataset for PET report generation with metabolic information, uniquely capturing whole-body image-report pairs that cover dozens of organs to fill the critical gap in existing benchmarks and mirror real-world clinical comprehensiveness. In addition to widely recognized natural language generation metrics, we introduce a series of clinical efficiency metrics to evaluate the quality of radiotracer uptake pattern description in key organs in generated reports. We conduct a head-to-head comparison of 30 cutting-edge general-purpose and medical-specialized VLMs. The results show that the current state-of-the-art VLMs perform poorly on PET report generation task, falling considerably short of fulfilling practical needs. Moreover, we identify several key insufficiency that need to be addressed to advance the development in medical applications.

Problem

• Automate PET radiology report generation using vision-language models
• Address lack of PET-specific datasets for metabolic imaging analysis
• Evaluate clinical relevance of AI-generated PET reports comprehensively

Innovation

• First dedicated PET report generation dataset
• Clinical efficiency metrics for radiotracer evaluation
• Comparison of 30 cutting-edge VLMs