ProtoTTA: Prototype-Guided Test-Time Adaptation

2026-04-16

AI Summary

This work addresses the limited robustness of existing test-time adaptation (TTA) methods under distribution shifts, which often fail to effectively leverage prototypical signals. To overcome this, we propose ProtoTTA, the first framework to integrate prototype mechanisms into TTA. ProtoTTA enhances model robustness by minimizing the entropy of prototype similarity distributions, thereby encouraging more confident and semantically focused prototype activations. Stability during adaptation is further ensured through geometric filtering and importance-weighted updating strategies. Evaluated across four prototypical backbones and four cross-modal benchmarks, ProtoTTA consistently outperforms state-of-the-art methods. Our results not only demonstrate improved robustness but also reveal a strong correlation between semantic focus and reasoning quality. Additionally, we introduce a novel interpretability metric and an evaluation framework tailored for vision-language models.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Deep networks that rely on prototypes-interpretable representations that can be related to the model input-have gained significant attention for balancing high accuracy with inherent interpretability, which makes them suitable for critical domains such as healthcare. However, these models are limited by their reliance on training data, which hampers their robustness to distribution shifts. While test-time adaptation (TTA) improves the robustness of deep networks by updating parameters and statistics, the prototypes of interpretable models have not been explored for this purpose. We introduce ProtoTTA, a general framework for prototypical models that leverages intermediate prototype signals rather than relying solely on model outputs. ProtoTTA minimizes the entropy of the prototype-similarity distribution to encourage more confident and prototype-specific activations on shifted data. To maintain stability, we employ geometric filtering to restrict updates to samples with reliable prototype activations, regularized by prototype-importance weights and model-confidence scores. Experiments across four prototypical backbones on four diverse benchmarks spanning fine-grained vision, histopathology, and NLP demonstrate that ProtoTTA improves robustness over standard output entropy minimization while restoring correct semantic focus in prototype activations. We also introduce novel interpretability metrics and a vision-language model (VLM) evaluation framework to explain TTA dynamics, confirming ProtoTTA restores human-aligned semantic focus and correlates reliably with VLM-rated reasoning quality. Code is available at: https://github.com/DeepRCL/ProtoTTA.

Problem

• prototype
• test-time adaptation
• distribution shift
• interpretability
• robustness

Innovation

• Prototype-Guided
• Test-Time Adaptation
• Interpretable Deep Learning
• Entropy Minimization
• Vision-Language Model