Watching Synthetic Videos: Aligning Cross-modal Representations with Visual Synthesis for Zero-shot Video Captioning

2026-08-11

AI Summary

This work addresses the cross-modal representation discrepancy in zero-shot video captioning, which arises from training solely on textual data. To bridge this gap, the authors propose a two-stage framework: first, leveraging a pretrained text-to-video model to generate latent representations of synthetic videos and enhancing their realism via a novel “polisher” module; second, designing a “prompter” module that conditions a GPT-2 language model on these refined latent representations to generate captions. By incorporating synthetic video latents and a distribution alignment mechanism, the approach overcomes the limitations of conventional linear alignment strategies, effectively mitigating the modality gap between training and inference. The method achieves state-of-the-art performance on MSVD, MSR-VTT, and VATEX, with BLEU@4 scores up to 52 and CIDEr scores reaching 95.7.

Abstract

Text-only training is a popular paradigm in zero-shot video captioning, where the video distribution is not available to the model during training, leading to a cross-modal gap between the training (text-only) and the inference (video-only). Previous works attempt to bridge the gap through simple linear transformations. However, the inherent gap between text and video makes cross-modal representation space alignment insufficient, resulting in inaccurate sentences. To address this issue, we propose a novel zero-shot video captioning framework (WSV) consisting of two training stages, which first generates corresponding synthetic video latent representations via a pretrained text-to-video generation model. To strengthen the fidelity of the latent representations, we propose a polisher capable of bridging the gap between real and synthetic video distributions. Subsequently, we design a prompter that conditions GPT-2 on the polished latent representations to generate the captions in the second training stage. During inference, an input video is encoded by a pretrained 3D Causal VAE and then fed directly into the prompter, which in turn guides GPT-2 to produce the final caption. Experimental results conducted on MSVD, MSR-VTT, and VATEX datasets demonstrate that our proposed method achieves scores of 52 and 95.7 on the B@4 and CIDEr metrics, respectively.

Problem

• zero-shot video captioning
• cross-modal gap
• text-only training
• video-text alignment
• synthetic video

Innovation

• zero-shot video captioning
• cross-modal alignment
• synthetic video generation
• latent representation polishing
• text-to-video prompting