🤖 AI Summary
Open-world video recognition suffers from significant domain shifts and poor generalization, while existing methods fail to effectively leverage the multimodal knowledge of foundation models. To address this, we propose PCA—a three-stage knowledge transfer framework: (1) *Percept*, which aligns video feature distributions via domain-aware visual alignment; (2) *Chat*, where a prompt-driven large language model (LLM) generates fine-grained semantic descriptions to construct textual knowledge; and (3) *Adapt*, employing a plug-and-play adapter module for cross-modal feature alignment and fusion. Our work introduces the first progressive multimodal knowledge distillation paradigm that jointly transfers visual and linguistic knowledge from frozen foundation models to a lightweight video recognizer—without any LLM fine-tuning. Evaluated on three open-world benchmarks—TinyVIRAT, ARID, and QV-Pipe—our method achieves state-of-the-art performance, with average accuracy improvements of 3.2–5.7 percentage points.
📝 Abstract
Open-world video recognition is challenging since traditional networks are not generalized well on complex environment variations. Alternatively, foundation models with rich knowledge have recently shown their generalization power. However, how to apply such knowledge has not been fully explored for open-world video recognition. To this end, we propose a generic knowledge transfer pipeline, which progressively exploits and integrates external multimodal knowledge from foundation models to boost open-world video recognition. We name it PCA, based on three stages of Percept, Chat, and Adapt. First, we perform Percept process to reduce the video domain gap and obtain external visual knowledge. Second, we generate rich linguistic semantics as external textual knowledge in Chat stage. Finally, we blend external multimodal knowledge in Adapt stage, by inserting multimodal knowledge adaptation modules into networks. We conduct extensive experiments on three challenging open-world video benchmarks, i.e., TinyVIRAT, ARID, and QV-Pipe. Our approach achieves state-of-the-art performance on all three datasets.