A Shared Encoder Approach to Multimodal Representation Learning

2025-03-03

AI Summary

Medical multimodal learning faces challenges including scarcity of paired data and reliance on proprietary or pretrained encoders. To address these, this paper proposes a single-encoder, parameter-sharing framework that unifies text and imaging modalities within a shared Transformer architecture. It introduces learnable modality embeddings for adaptive representation learning and designs a cross-modal parameter-sharing mechanism coupled with a joint contrastive alignment loss to alleviate low-resource generalization bottlenecks. Crucially, the approach eliminates modality-specific encoders. Evaluated across multiple medical multimodal benchmarks, it achieves significant improvements in few-shot settings (<1k samples): average retrieval accuracy increases by 4.2%, and classification F1 score improves by 3.8%. The core contribution is the first lightweight, parameter-shared multimodal representation learning paradigm explicitly designed for low-resource medical scenarios.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multimodal representation learning has demonstrated remarkable potential in enabling models to process and integrate diverse data modalities, such as text and images, for improved understanding and performance. While the medical domain can benefit significantly from this paradigm, the scarcity of paired multimodal data and reliance on proprietary or pretrained encoders pose significant challenges. In this work, we present a shared encoder framework for multimodal representation learning tailored to the medical domain. Our approach employs a single set of encoder parameters shared across modalities, augmented with learnable modality features. Empirical results demonstrate that our shared encoder idea achieves superior performance compared to separate modality-specific encoders, demonstrating improved generalization in data-constrained settings. Notably, the performance gains are more pronounced with fewer training examples, underscoring the efficiency of our shared encoder framework for real-world medical applications with limited data. Our code and experiment setup are available at https://github.com/VectorInstitute/shared_encoder.

Problem

• Addresses scarcity of paired multimodal data in medical domain.
• Proposes shared encoder framework for multimodal representation learning.
• Improves generalization with limited training data in medical applications.

Innovation

• Shared encoder for multimodal learning
• Single encoder with modality features
• Improved generalization with limited data