DPCore: Dynamic Prompt Coreset for Continual Test-Time Adaptation

2024-06-15

AI Summary

This work addresses convergence difficulties, catastrophic forgetting, and erroneous knowledge transfer in continual test-time adaptation (CTTA), caused by non-stationary target-domain shifts—varying in frequency and duration. To this end, we propose the first lightweight adaptive prompting method tailored for realistic dynamic environments. Our core innovation is a dynamic prompt core-set mechanism: it adaptively reuses or generates visual prompts based on similarity criteria and employs a lightweight learnable prompt encoder for parameter-efficient updates. Evaluated on four standard CTTA benchmarks, our method achieves state-of-the-art performance in both structured and unstructured dynamic scenarios. It reduces model parameters by 99% and inference latency by 64% compared to existing approaches, while significantly improving generalization robustness and adaptation efficiency.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Continual Test-Time Adaptation (CTTA) seeks to adapt source pre-trained models to continually changing, unseen target domains. While existing CTTA methods assume structured domain changes with uniform durations, real-world environments often exhibit dynamic patterns where domains recur with varying frequencies and durations. Current approaches, which adapt the same parameters across different domains, struggle in such dynamic conditions-they face convergence issues with brief domain exposures, risk forgetting previously learned knowledge, or misapplying it to irrelevant domains. To remedy this, we propose DPCore, a method designed for robust performance across diverse domain change patterns while ensuring computational efficiency. DPCore integrates three key components: Visual Prompt Adaptation for efficient domain alignment, a Prompt Coreset for knowledge preservation, and a Dynamic Update mechanism that intelligently adjusts existing prompts for similar domains while creating new ones for substantially different domains. Extensive experiments on four benchmarks demonstrate that DPCore consistently outperforms various CTTA methods, achieving state-of-the-art performance in both structured and dynamic settings while reducing trainable parameters by 99% and computation time by 64% compared to previous approaches.

Problem

• Adapts models to changing domains
• Handles dynamic domain recurrence
• Reduces computational resources required

Innovation

• Dynamic Prompt Coreset
• Visual Prompt Adaptation
• Dynamic Update mechanism