OncoVision: Integrating Mammography and Clinical Data through Attention-Driven Multimodal AI for Enhanced Breast Cancer Diagnosis

2025-11-24

AI Summary

This study addresses three critical challenges in breast cancer diagnosis: insufficient integration of imaging and clinical data, substantial inter-observer variability, and the lack of accessible screening tools for primary-care settings. We propose an attention-driven multimodal AI system. Methodologically, it employs a dual-path late-fusion architecture that jointly processes mammographic images and structured clinical variables; an attention-based encoder-decoder model enables simultaneous segmentation of four key lesion regions; and dual confidence scoring with attention visualization enhances model interpretability. A lightweight web platform facilitates real-time clinical decision support. Contributions include: (1) significantly improved diagnostic accuracy—achieving state-of-the-art performance—and enhanced reader agreement, reducing inter-observer variability; (2) generation of standardized, structured reports with human-verifiable explanations, thereby strengthening clinical trust; and (3) high scalability and deployment feasibility, enabling large-scale early screening and medical education in resource-limited settings.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Algorithmic accountability and transparency on the web

Abstract

OncoVision is a multimodal AI pipeline that combines mammography images and clinical data for better breast cancer diagnosis. Employing an attention-based encoder-decoder backbone, it jointly segments four ROIs - masses, calcifications, axillary findings, and breast tissues - with state-of-the-art accuracy and robustly predicts ten structured clinical features: mass morphology, calcification type, ACR breast density, and BI-RADS categories. To fuse imaging and clinical insights, we developed two late-fusion strategies. By utilizing complementary multimodal data, late fusion strategies improve diagnostic precision and reduce inter-observer variability. Operationalized as a secure, user-friendly web application, OncoVision produces structured reports with dual-confidence scoring and attention-weighted visualizations for real-time diagnostic support to improve clinician trust and facilitate medical teaching. It can be easily incorporated into the clinic, making screening available in underprivileged areas around the world, such as rural South Asia. Combining accurate segmentation with clinical intuition, OncoVision raises the bar for AI-based mammography, offering a scalable and equitable solution to detect breast cancer at an earlier stage and enhancing treatment through timely interventions.

Problem

• Integrating mammography images with clinical data for enhanced breast cancer diagnosis
• Segmenting four key breast tissue regions and predicting ten clinical features
• Reducing diagnostic variability and improving screening accessibility in underserved areas

Innovation

• Attention-based multimodal AI for mammography and clinical data
• Late-fusion strategies integrating imaging and clinical insights
• Web application generating structured reports with confidence scoring