Keeping Representation Similarity in Finetuning for Medical Image Analysis

2025-03-10

AI Summary

In medical image fine-tuning, pretrained foundation model representations often degenerate, undermining generalization. To address this, we propose RepSim—a novel method that introduces, for the first time, a learnable, similarity-invariant orthogonal manifold constraint to jointly optimize task adaptability and pretrained representation fidelity during fine-tuning. RepSim integrates orthogonal manifold optimization, representation similarity regularization, and contrastive distance minimization. Evaluated on five medical image classification benchmarks, it improves representation similarity by over 30%, reduces model sharpness by 42%, and maintains competitive classification accuracy. Our core contributions are: (1) a representation-similarity-driven manifold constraint paradigm that preserves structural invariance in latent space; and (2) a unified optimization objective that simultaneously retains pretrained knowledge and enhances downstream performance. RepSim thus bridges the gap between representation stability and task-specific adaptation in medical imaging.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Representation Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Foundation models pretrained on large-scale natural images have been widely used to adapt to medical image analysis through finetuning. This is largely attributed to pretrained representations capturing universal, robust, and generalizable features, which can be reutilized by downstream tasks. However, these representations are later found to gradually vanish during finetuning, accompanied by a degradation of foundation model's original abilities, e.g., generalizability. In this paper, we argue that pretrained representations can be well preserved while still effectively adapting to downstream tasks. We study this by proposing a new finetuning method RepSim, which minimizes the distance between pretrained and finetuned representations via constraining learnable orthogonal manifold based on similarity invariance. Compared to standard finetuning methods, e.g., full finetuning, our method improves representation similarity by over 30% while maintaining competitive accuracy, and reduces sharpness by 42% across five medical image classification datasets. The code will be released.

Problem

• Preserve pretrained representations during finetuning for medical image analysis.
• Prevent degradation of foundation model's generalizability in downstream tasks.
• Improve representation similarity and reduce sharpness in medical image classification.

Innovation

• Preserves pretrained representations during finetuning
• Uses orthogonal manifold for similarity invariance
• Improves representation similarity by over 30%