Semantic Alignment of Unimodal Medical Text and Vision Representations

2025-03-06

AI Summary

To address the suboptimal performance of general-purpose AI models on medical imaging tasks, this paper proposes a fine-tuning-free semantic alignment framework for zero-shot knowledge transfer from general multimodal models to the medical domain. Methodologically, we introduce a lightweight anchor-based affine transformation mechanism that aligns general-purpose text and visual representations into a unified semantic space. We further pioneer a zero-shot medical classification paradigm tailored for unimodal visual encoders, integrating semantic anchor matching, cross-modal representation alignment, and model stitching. Evaluated on multiple public chest X-ray datasets, our method achieves zero-shot inference performance surpassing state-of-the-art general multimodal models and approaching that of fully supervised, domain-specific medical models—without requiring any task-specific training data or parameter updates. This significantly reduces the cost and complexity of domain adaptation while enabling robust, scalable deployment of foundation models in clinical imaging applications.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

General-purpose AI models, particularly those designed for text and vision, demonstrate impressive versatility across a wide range of deep-learning tasks. However, they often underperform in specialised domains like medical imaging, where domain-specific solutions or alternative knowledge transfer approaches are typically required. Recent studies have noted that general-purpose models can exhibit similar latent spaces when processing semantically related data, although this alignment does not occur naturally. Building on this insight, it has been shown that applying a simple transformation - at most affine - estimated from a subset of semantically corresponding samples, known as anchors, enables model stitching across diverse training paradigms, architectures, and modalities. In this paper, we explore how semantic alignment - estimating transformations between anchors - can bridge general-purpose AI with specialised medical knowledge. Using multiple public chest X-ray datasets, we demonstrate that model stitching across model architectures allows general models to integrate domain-specific knowledge without additional training, leading to improved performance on medical tasks. Furthermore, we introduce a novel zero-shot classification approach for unimodal vision encoders that leverages semantic alignment across modalities. Our results show that our method not only outperforms general multimodal models but also approaches the performance levels of fully trained, medical-specific multimodal solutions

Problem

• Bridging general-purpose AI with specialized medical knowledge.
• Improving medical task performance without additional training.
• Introducing zero-shot classification for unimodal vision encoders.

Innovation

• Semantic alignment bridges general AI with medical knowledge.
• Model stitching integrates domain-specific knowledge without retraining.
• Zero-shot classification leverages semantic alignment across modalities.