F^2TTA: Free-Form Test-Time Adaptation on Cross-Domain Medical Image Classification via Image-Level Disentangled Prompt Tuning

2025-07-03

AI Summary

To address the challenge of clinical medical imaging test data arriving as unlabeled, variable-length, and randomly ordered domain fragments—with unpredictable distribution shifts between fragments—this paper introduces Free-form Test-Time Adaptation (F2TTA), the first formalization of this problem. To mitigate distribution shift and catastrophic forgetting, we propose an image-level disentangled prompt tuning framework featuring: (i) invariant–specific dual-path prompts, (ii) an uncertainty-driven dynamic masking mechanism, and (iii) parallel graph networks for historical knowledge distillation. Our method enables efficient, single-image-granularity adaptation without requiring access to past data or labels. Evaluated on breast cancer and glaucoma classification tasks, it significantly outperforms existing test-time adaptation methods, demonstrating robustness and practicality in real-world fragmented medical data streams. The implementation is publicly available.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Security and Privacy: Data transparency and provenance

Abstract

Test-Time Adaptation (TTA) has emerged as a promising solution for adapting a source model to unseen medical sites using unlabeled test data, due to the high cost of data annotation. Existing TTA methods consider scenarios where data from one or multiple domains arrives in complete domain units. However, in clinical practice, data usually arrives in domain fragments of arbitrary lengths and in random arrival orders, due to resource constraints and patient variability. This paper investigates a practical Free-Form Test-Time Adaptation (F$^{2}$TTA) task, where a source model is adapted to such free-form domain fragments, with shifts occurring between fragments unpredictably. In this setting, these shifts could distort the adaptation process. To address this problem, we propose a novel Image-level Disentangled Prompt Tuning (I-DiPT) framework. I-DiPT employs an image-invariant prompt to explore domain-invariant representations for mitigating the unpredictable shifts, and an image-specific prompt to adapt the source model to each test image from the incoming fragments. The prompts may suffer from insufficient knowledge representation since only one image is available for training. To overcome this limitation, we first introduce Uncertainty-oriented Masking (UoM), which encourages the prompts to extract sufficient information from the incoming image via masked consistency learning driven by the uncertainty of the source model representations. Then, we further propose a Parallel Graph Distillation (PGD) method that reuses knowledge from historical image-specific and image-invariant prompts through parallel graph networks. Experiments on breast cancer and glaucoma classification demonstrate the superiority of our method over existing TTA approaches in F$^{2}$TTA. Code is available at https://github.com/mar-cry/F2TTA.

Problem

• Adapting source models to unpredictable domain fragments in medical images
• Mitigating shifts between domain fragments during test-time adaptation
• Enhancing knowledge representation with limited test images for adaptation

Innovation

• Image-level Disentangled Prompt Tuning (I-DiPT) framework
• Uncertainty-oriented Masking (UoM) for consistency learning
• Parallel Graph Distillation (PGD) reusing historical prompts