A multimodal vision foundation model for generalizable knee pathology

2026-01-26

AI Summary

This work addresses the limitations of existing orthopedic AI models, which are typically task-specific, reliant on large amounts of annotated data, and lack generalization across modalities and anatomical sites. To overcome these challenges, the authors construct a large-scale pretraining dataset comprising 1.2 million unannotated knee X-ray and MRI images and develop OrthoFoundation, a multimodal vision foundation model trained via self-supervised contrastive learning using the DINOv3 backbone. This study presents the first large-scale multimodal self-supervised pretraining framework in musculoskeletal imaging, demonstrating strong transferability to other anatomical regions such as the hip, shoulder, and ankle. OrthoFoundation achieves state-of-the-art performance across 14 downstream tasks, outperforming existing methods in X-ray-based osteoarthritis diagnosis and MRI-based structural damage detection, while matching fully supervised models with only 50% of the labeled data.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Other Foundations of Computer Vision
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Musculoskeletal disorders represent a leading cause of global disability, creating an urgent demand for precise interpretation of medical imaging. Current artificial intelligence (AI) approaches in orthopedics predominantly rely on task-specific, supervised learning paradigms. These methods are inherently fragmented, require extensive annotated datasets, and often lack generalizability across different modalities and clinical scenarios. The development of foundation models in this field has been constrained by the scarcity of large-scale, curated, and open-source musculoskeletal datasets. To address these challenges, we introduce OrthoFoundation, a multimodal vision foundation model optimized for musculoskeletal pathology. We constructed a pre-training dataset of 1.2 million unlabeled knee X-ray and MRI images from internal and public databases. Utilizing a Dinov3 backbone, the model was trained via self-supervised contrastive learning to capture robust radiological representations. OrthoFoundation achieves state-of-the-art (SOTA) performance across 14 downstream tasks. It attained superior accuracy in X-ray osteoarthritis diagnosis and ranked first in MRI structural injury detection. The model demonstrated remarkable label efficiency, matching supervised baselines using only 50% of labeled data. Furthermore, despite being pre-trained on knee images, OrthoFoundation exhibited exceptional cross-anatomy generalization to the hip, shoulder, and ankle. OrthoFoundation represents a significant advancement toward general-purpose AI for musculoskeletal imaging. By learning fundamental, joint-agnostic radiological semantics from large-scale multimodal data, it overcomes the limitations of conventional models, which provides a robust framework for reducing annotation burdens and enhancing diagnostic accuracy in clinical practice.

Problem

• musculoskeletal disorders
• foundation models
• medical imaging
• generalizability
• annotation scarcity

Innovation

• multimodal foundation model
• self-supervised contrastive learning
• cross-anatomy generalization
• label efficiency
• musculoskeletal imaging