GoTTA be Diverse: Rethinking Memory Policies for Test-Time Adaptation

2026-05-19

AI Summary

This work addresses the lack of systematic evaluation of memory mechanisms in existing test-time adaptation (TTA) methods, which obscures the identification of key factors underlying effective strategies. By decoupling memory design from adaptation algorithms, we establish a unified framework to evaluate diverse test-stream scenarios and propose a novel paradigm centered on intra-class diversity. We reveal, for the first time, that intra-class diversity is critical for avoiding redundant buffering and preserving representative adaptive signals. Building on this insight, we introduce GOTTA, a plug-and-play, diversity-aware memory strategy that integrates class-balanced allocation with feature-space diversity and is compatible with various TTA objective functions. Experiments demonstrate that GOTTA significantly improves performance under limited memory and challenging non-i.i.d. streaming conditions, maintaining competitiveness even as memory capacity increases, thereby validating intra-class diversity as a fundamental principle in TTA.

Abstract

Test-time adaptation (TTA) enables a pre-trained model to adapt online to an unlabeled test stream under distribution shift. While most TTA research focuses on the adaptation objective, practical streams also depend critically on the memory used to select which test samples drive adaptation. Existing memory mechanisms are usually evaluated as components of specific TTA algorithms, making it difficult to isolate which memory design choices matter and when they matter. In this work, we provide a systematic benchmark that decouples memory from the adaptation algorithm and evaluates memory policies under unified conditions across i.i.d., non-i.i.d., continual, and practical test streams. Our study shows that effective memory management requires more than retaining recent or class-balanced samples. In particular, intra-class diversity is a key factor for avoiding redundant buffers and maintaining representative adaptation signals under temporally correlated and label-skewed streams. Motivated by this finding, we introduce Guided Observational Test-Time Adaptation (GOTTA), a family of diversity-aware memory policies that combine class-balanced allocation with feature-space diversity. GOTTA memories act as drop-in replacements for existing buffers and can be paired with different TTA objectives. Across corruption benchmarks and video-stream settings, diversity-aware memory improves adaptation most clearly under constrained memory budgets and challenging non-i.i.d. streams, while remaining competitive as memory capacity increases. These results highlight memory management as a first-class component of robust test-time adaptation and identify diversity as a central principle for practical TTA.

Problem

• test-time adaptation
• memory policy
• distribution shift
• non-i.i.d. streams
• intra-class diversity

Innovation

• test-time adaptation
• memory policy
• intra-class diversity
• non-i.i.d. streams
• GOTTA