KeyFrame-Compass: Towards Comprehensive Evaluation of Keyframe-Conditioned Video Generation

2026-07-15

AI Summary

Existing keyframe-conditioned video generation models lack systematic evaluation regarding their ability to faithfully reproduce specified keyframes while maintaining overall video quality. This work introduces the first comprehensive benchmark for this task, encompassing diverse scenes, structural configurations, and conditioning formats, along with an automated evaluation framework. The framework comprises a six-dimensional metric assessing keyframe adherence—covering existence, fidelity, temporal alignment, spatial localization, persistence, and uniqueness—and integrates both multimodal large language models and specialized perceptual models to evaluate video quality. Experiments across nine state-of-the-art systems reveal a consistent trade-off between keyframe fidelity and visual naturalness, with performance notably degrading under dense keyframe sequences or storyboard-style inputs.

Abstract

Video generation increasingly relies on keyframe-based workflows, where creators specify a sequence of reference images to guide generation. Although recent models support multi-keyframe conditioning, it remains unclear whether they can faithfully reproduce the prescribed keyframes while maintaining overall video quality. We present KeyFrame-Compass, the first comprehensive benchmark for evaluating keyframe-conditioned video generation. The benchmark contains 386 carefully curated samples spanning three application domains, two video structures, two prompt granularities, two conditioning formats, and four keyframe densities, enabling controlled analysis under diverse generation settings. We further introduce an automated evaluation framework that jointly measures keyframe execution and overall video quality. Specifically, we decompose keyframe execution into six complementary metrics covering presence, fidelity, temporal ordering, localization, persistence, and uniqueness, while assessing overall video quality through evidence-grounded MLLM judgments augmented with specialized perception models. Experiments on nine representative video generation systems reveal several fundamental limitations. Current models exhibit a clear trade-off between faithful keyframe execution and natural video synthesis. Their performance further degrades as keyframe constraints become denser and most open-source models also fail to interpret storyboard-grid inputs as temporally ordered keyframe sequences.

Problem

• keyframe-conditioned video generation
• video quality
• keyframe fidelity
• evaluation benchmark
• temporal consistency

Innovation

• keyframe-conditioned video generation
• comprehensive benchmark
• automated evaluation framework
• temporal consistency
• video quality assessment