SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation

2026-08-18

AI Summary

本文提出了一种以结果为导向的视频生成任务SemComp-Bench，通过构建包含六个领域的评估数据集SemComp-Data，并使用视觉-语言模型进行结构化二元问题回答来评价生成视频的结果达成度和生成可靠性。

Abstract

We introduce Semantic Task Completion Video Generation, an outcome-oriented video generation task. Under this formulation, success requires both achievement of the intended outcome and semantic grounding. Semantic grounding characterizes the correspondence between the reference image and the generated outcome in terms of high-level semantics relevant to the task. Evaluation focuses on the generated outcome and requires neither the presentation of a complete sequence of intermediate task steps nor conventional appearance consistency with the reference image. To support systematic evaluation, we construct SemComp-Data, an evaluation dataset covering six domains. Each instance comprises a reference image, a detailed instruction, a brief instruction, and an outcome-centric video clip. A scalable four-stage curation pipeline converts raw videos into standardized SemComp-Data instances. We further introduce SemComp-Bench, an evaluation protocol that uses a vision-language model (VLM) to answer structured binary questions. SemComp-Bench reports the OA Score and the GR Score for Outcome Achievement and Generation Reliability, respectively. Experiments on representative video generation models show that achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging.

Problem

• Semantic Task Completion
• Video Generation
• Outcome Achievement
• Semantic Grounding
• Reference Image

Innovation

• Semantic Task Completion
• Outcome-oriented Video Generation
• SemComp-Data
• SemComp-Bench
• Vision-Language Model