TACO: Towards Task-Consistent Open-Vocabulary Adaptation in Video Recognition

2026-06-24

AI Summary

This work addresses the degradation in out-of-distribution generalization of CLIP for open-vocabulary video recognition, which arises from the misalignment between fine-tuning and evaluation objectives. To resolve this issue, the authors propose a task-consistent adaptation method that decouples the representation space from the optimization space through a relative structural distillation mechanism and a lightweight specialized projection head. This design preserves semantic alignment consistency under distribution shifts while mitigating alignment drift during training. The proposed approach achieves state-of-the-art performance across multiple benchmark settings—including cross-dataset transfer and base-to-novel class generalization—demonstrating significantly enhanced open-vocabulary generalization capabilities.

Abstract

Adapting CLIP for open-vocabulary video recognition necessitates a delicate balance between newly acquired video knowledge and the pretrained generalization. While existing studies pursue this generalization-specialization trade-off with additional regularizations or constraints, we argue that they overlook the deviation of representations beyond the fine-tuning data distribution, resulting in suboptimal adaptation effects. We believe such deviation is inherited from the inconsistency between the fine-tuning and evaluation objectives, where model optimization is restricted to the known training distribution but evaluated on unseen ones. In this paper, we introduce \emph{TACO}, a simple yet effective framework to mitigate the potential negative effects induced by this inconsistency. Our key insight is that adaptation should preserve OOD-relevant alignment beyond the training distribution. To this end, we propose \emph{Relative Structure Distillation}, which regularizes the relative geometry of the representation space and suppresses harmful alignment shift during training. We further decouple the representation space from the optimization space with a lightweight specialization projection, allowing task-specific adaptation without directly overspecializing the representations used at test time. \emph{TACO} establishes state-of-the-art performance on diverse benchmarks under cross-dataset and base-to-novel settings. Code will be released at https://github.com/ZMHH-H/TACO.

Problem

• open-vocabulary video recognition
• task inconsistency
• out-of-distribution generalization
• representation alignment
• model adaptation

Innovation

• Task-Consistent Adaptation
• Open-Vocabulary Video Recognition
• Relative Structure Distillation
• Representation Decoupling
• Out-of-Distribution Generalization