DirectorBench: Diagnosing Long-Form Video Generation with Personalized Multi-Agent Evaluation

2026-05-28

AI Summary

Existing evaluation methods for long-form video generation struggle to provide fine-grained diagnostics of narrative structure, multimodal synchronization, and user preference alignment, while also lacking in-depth analysis of workflow bottlenecks and personalization requirements. This work proposes the first multi-agent diagnostic evaluation framework that incorporates user-profile awareness and checkpoint-level bottleneck localization. By integrating 80 structured metadata attributes, 7 user personas, and 40 cross-modal checkpoints, the framework overcomes the limitations of conventional single-aggregate scoring. Experiments across four generation workflows and six large language models demonstrate its ability to effectively uncover human-perceivable quality discrepancies: transition quality scores averaged only 0.256, whereas user-need satisfaction reached 0.71. Human evaluations corroborate the framework’s diagnostic validity and sensitivity.

Abstract

Long-form video generation is rapidly moving from short, single-scene synthesis toward minute-long, multi-shot creation with narrative structure, cinematic control, audio, and cross-modal synchronization. However, evaluating such videos remains challenging, since existing benchmarks largely focus on local visual quality, short-horizon temporal consistency, or generic prompt alignment, and provide limited diagnosis of workflow failures and user-dependent preferences. We introduce DirectorBench, a personalized multi-agent diagnostic benchmark for long-form video generation. DirectorBench evaluates generated videos with respect to 80 structured metadata entries, 7 user profiles, and 40 checkpoint criteria across 5 dimensions: script, visual, audio, cross-modal, and stability. Instead of reducing quality to a single aggregate score, DirectorBench localizes checkpoint-level bottlenecks and supports profile-aware evaluation. We evaluate 4 long-form video generation workflows, 6 base LLMs, and 7 user profiles. Across workflows, DirectorBench reveals a between-unit bottleneck: transition quality averages only 0.256 and reaches 0.356 for the best workflow, while prompt-level user demand fulfillment averages 0.71. We further conduct human evaluation with 14 annotators to validate the alignment between DirectorBench and human judgment. The results show that DirectorBench captures human-perceptible quality differences and reveals workflow- and profile-dependent failure modes that are hidden by aggregate scoring. These findings highlight the importance of diagnostic and profile-aware benchmarking for long-form video generation.

Problem

• long-form video generation
• evaluation benchmark
• personalized assessment
• multi-agent diagnosis
• workflow failure

Innovation

• long-form video generation
• personalized evaluation
• multi-agent benchmarking
• diagnostic assessment
• cross-modal synchronization