Segmentation Assisted Incremental Test Time Adaptation in an Open World

2025-08-27

AI Summary

This paper addresses the Incremental Test-Time Adaptation (ITTA) problem for vision-language models (VLMs) in dynamic open-world settings—where models encounter unseen classes and joint covariate and label distribution shifts during testing. We propose the first zero-shot test-time learning framework supporting both class-incremental learning and distribution adaptation. Our core innovation is SegAssist, a training-free segmentation-assisted active annotation module that uniquely leverages the VLM’s intrinsic zero-shot segmentation capability to prioritize identification and pseudo-labeling of potential novel-class samples. Integrated with single-image test-time adaptation and an oracle-query mechanism, SegAssist enables a closed-loop process of novel-class discovery, annotation, and model update. Extensive experiments on multiple benchmarks demonstrate significant improvements in cross-domain generalization and novel-class recognition. Our approach establishes a scalable, fine-tuning-free paradigm for open-world continual learning.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

In dynamic environments, unfamiliar objects and distribution shifts are often encountered, which challenge the generalization abilities of the deployed trained models. This work addresses Incremental Test Time Adaptation of Vision Language Models, tackling scenarios where unseen classes and unseen domains continuously appear during testing. Unlike traditional Test Time Adaptation approaches, where the test stream comes only from a predefined set of classes, our framework allows models to adapt simultaneously to both covariate and label shifts, actively incorporating new classes as they emerge. Towards this goal, we establish a new benchmark for ITTA, integrating single image TTA methods for VLMs with active labeling techniques that query an oracle for samples potentially representing unseen classes during test time. We propose a segmentation assisted active labeling module, termed SegAssist, which is training free and repurposes the segmentation capabilities of VLMs to refine active sample selection, prioritizing samples likely to belong to unseen classes. Extensive experiments on several benchmark datasets demonstrate the potential of SegAssist to enhance the performance of VLMs in real world scenarios, where continuous adaptation to emerging data is essential. Project-page:https://manogna-s.github.io/segassist/

Problem

• Adapting vision language models to unseen classes and domains during testing
• Handling both covariate and label shifts in dynamic open-world environments
• Enabling continuous incremental adaptation without predefined class constraints

Innovation

• Segmentation assisted active labeling module
• Training free repurposing VLM segmentation
• Continuous adaptation to emerging data