Zero-shot video highlight detection based on text descriptions and synthetic images

2026-09-13

AI Summary

该研究提出了一种基于文本描述和合成图像的零样本视频高光检测框架，结合了CLIP、大语言模型和扩散模型来解决无需标注数据的视频高光检测问题。

Abstract

Detecting video highlights, the most informative or engaging moments in a video, is important for applications such as video summarization and content recommendation. We propose a zero-shot framework that combines CLIP, large language models (LLMs), and diffusion models. Given lightweight video metadata, such as a title or category, an LLM generates textual descriptions of likely highlight events. These descriptions are further converted into synthetic visual prototypes using a diffusion model. Textual and visual representations are matched to video frames using CLIP, enabling frame-level highlight detection without highlight annotations or dataset-specific training. Experiments on TVSum and SumMe demonstrate strong zero-shot performance, with particularly favorable results on TVSum. The proposed approach provides an effective framework for metadata-conditioned zero-shot video highlight detection.

Problem

• zero-shot
• video highlight detection
• text descriptions
• synthetic images

Innovation

• zero-shot framework
• CLIP
• large language models (LLMs)
• diffusion models
• metadata-conditioned