Towards Continual Test-Time Adaptation of Vision-Language Models in Open-Vocabulary Semantic Segmentation

2026-08-30

AI Summary

为解决开放词汇语义分割中持续测试时分布偏移导致的视觉-语言对齐脆弱问题，提出DAF框架，通过增加边缘多样性损失、跨模态锚点一致性损失和特征显著性过滤来稳定模型。

Abstract

Open-vocabulary semantic segmentation (OVSS) relies on vision-language alignment to recognize arbitrary text-defined categories, yet this alignment is fragile under continual test-time distribution shift. Our diagnostic analysis reveals that entropy minimization drives patch-level class collapse, continual updates erode vision-language alignment, and redundant gradients from low-shift samples waste computation. We propose Diversify, Anchor, and Filter (DAF), a stabilization framework that augments entropy-based adaptation with a marginal diversity loss that resists collapse, a cross-modal anchor consistency loss that constrains feature drift relative to a frozen source model, and feature salience filtering that skips low-value backward passes to offset part of the source-anchor overhead. We evaluate on five datasets spanning natural scenes, autonomous driving, underwater imagery, and remote sensing with their corrupted variants. Across the evaluated continual shifts, DAF remains stable where entropy minimization collapses, improving mIoU by over 8 points on Pascal VOC20-C, over 9 points on LoveDA, and over 3 points on Foggy Cityscapes compared to the source model, and is robust to aggressive adaptation and learning rate choices.

Problem

• Open-vocabulary Semantic Segmentation
• Test-time Adaptation
• Vision-language Alignment
• Continual Distribution Shift

Innovation

• entropy minimization
• cross-modal anchor consistency loss
• feature salience filtering