TDVE-Assessor: Benchmarking and Evaluating the Quality of Text-Driven Video Editing with LMMs

2025-05-26

AI Summary

Text-driven video editing lacks dedicated video quality assessment (VQA) models, hindering rigorous evaluation. To address this, we introduce TDVE-DB—the first benchmark tailored to this task—comprising 3,857 videos and over 170,000 human annotations. We propose a three-dimensional evaluation framework: editing quality, text-video alignment, and structural consistency. Furthermore, we design TDVE-Assessor, the first specialized VQA model for text-driven editing, featuring a novel multimodal architecture that synergistically integrates spatiotemporal video encoders (e.g., VideoMAE) with fine-tuned large language models (LLMs) for fine-grained, semantics-aware assessment. Extensive experiments demonstrate that TDVE-Assessor consistently outperforms existing VQA methods across all three dimensions on TDVE-DB. Both the benchmark and model will be publicly released to foster standardized evaluation and advancement in the field.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Text-driven video editing is rapidly advancing, yet its rigorous evaluation remains challenging due to the absence of dedicated video quality assessment (VQA) models capable of discerning the nuances of editing quality. To address this critical gap, we introduce TDVE-DB, a large-scale benchmark dataset for text-driven video editing. TDVE-DB consists of 3,857 edited videos generated from 12 diverse models across 8 editing categories, and is annotated with 173,565 human subjective ratings along three crucial dimensions, i.e., edited video quality, editing alignment, and structural consistency. Based on TDVE-DB, we first conduct a comprehensive evaluation for the 12 state-of-the-art editing models revealing the strengths and weaknesses of current video techniques, and then benchmark existing VQA methods in the context of text-driven video editing evaluation. Building on these insights, we propose TDVE-Assessor, a novel VQA model specifically designed for text-driven video editing assessment. TDVE-Assessor integrates both spatial and temporal video features into a large language model (LLM) for rich contextual understanding to provide comprehensive quality assessment. Extensive experiments demonstrate that TDVE-Assessor substantially outperforms existing VQA models on TDVE-DB across all three evaluation dimensions, setting a new state-of-the-art. Both TDVE-DB and TDVE-Assessor will be released upon the publication.

Problem

• Lack of dedicated VQA models for text-driven video editing quality assessment
• Need for a benchmark dataset to evaluate diverse video editing models
• Developing a specialized VQA model integrating spatial and temporal features

Innovation

• Large-scale benchmark dataset for video editing
• VQA model integrating spatial-temporal features
• Quality assessment via large language model