Robust Zero-Shot Generalization for Open-Vocabulary Action Recognition via Task Arithmetic

2026-06-17

AI Summary

This work addresses the challenge of open-vocabulary action recognition in real-world scenarios, where robust zero-shot generalization is hindered by the need for costly and privacy-sensitive domain-specific fine-tuning. To overcome this limitation, the authors propose a novel paradigm that eliminates the requirement for target-domain adaptation. Their approach leverages task arithmetic to extract task vectors from models fine-tuned on multiple public open-vocabulary action recognition (OVAR) datasets, then recombines these vectors through model merging guided by vision-language representations to enable cross-domain knowledge transfer. Evaluated under out-of-distribution settings, the resulting merged model significantly improves zero-shot action recognition performance and achieves higher accuracy than the original pre-trained model without any target-domain fine-tuning.

Abstract

Open Vocabulary Action Recognition (OVAR) enables the recognition of novel actions by leveraging vision-language representations, overcoming the limitations of traditional closed-set approaches. However, achieving robust performance in real-world scenarios typically requires domain-specific fine-tuning, which is often costly and raises privacy and regulatory concerns. In this work, we propose an alternative paradigm that bypasses target-domain training and recombines knowledge from existing datasets and models. Leveraging model merging and task arithmetic, we extract and combine task vectors from models fine-tuned on diverse public OVAR datasets. We show that, in out-of-distribution settings, the resulting merged model achieves superior zero-shot generalization to the pre-trained base model. Code is available at https://github.com/omaymaMoussadek/robust-ovar

Problem

• Open-Vocabulary Action Recognition
• Zero-Shot Generalization
• Out-of-Distribution
• Domain Adaptation
• Vision-Language Representation

Innovation

• task arithmetic
• model merging
• zero-shot generalization
• open-vocabulary action recognition
• out-of-distribution