ResNetVLLM -- Multi-modal Vision LLM for the Video Understanding Task
For zero-shot video understanding, this paper proposes a cross-modal framework that eliminates the need for pre-trained video models. The method employs an off-the-shelf, non-video-pretrained ResNet as the visual encoder, directly interfaced with a large language model (LLM), and performs end-to-end joint optimization to align visual and linguistic representations. Its core contribution lies in departing from conventional video-specific pretraining paradigms: it pioneers the use of a frozen ResNet backbone—unfined on video data—combined with zero-shot prompt learning and cross-modal feature mapping, achieving superior generalization while preserving architectural simplicity. Experiments demonstrate state-of-the-art zero-shot performance on four standard benchmarks: MSRVTT-QA, MSVD-QA, TGIF-QA FrameQA, and ActivityNet-QA.