What Semantics Survive the Connector? Diagnosing VLM-to-DiT Alignment in Video Editing

2026-05-20

AI Summary

This work challenges the prevailing assumption in video editing that visual-language models (VLMs) and Diffusion Transformers (DiTs) can achieve lossless semantic alignment, an unverified premise that may compromise fine-grained structural semantics. To investigate this, the authors introduce a controllable video synthesis pipeline, along with TRACE-Edit—a diagnostic dataset—and a systematic evaluation protocol. Their analysis reveals, for the first time, significant semantic bottlenecks in the VLM-to-DiT alignment process, thereby refuting the hypothesis of lossless semantic transfer. By examining the impact of connector designs and meta-query mechanisms on semantic preservation, comprehensive evaluations across four representative models demonstrate that alignment substantially degrades structural semantics, highlighting the connector as a critical bottleneck. This study establishes the first diagnostic framework tailored for relational video editing.

Abstract

Flow matching based video generative models have been increasingly relying on prepended Vision-Language Models (VLMs) to handle complex, instruction-based video editing. The prevailing assumption underlying this paradigm is that a connector module can seamlessly align the VLM's rich multi-modal reasoning with the original text embedding space of DiTs. However, we hypothesize that this alignment acts as a severe semantic bottleneck, degrading fine-grained structural variables. Verifying this is challenging, as end-to-end evaluations conflate alignment failures with generation errors, and natural datasets lack disentangled annotations. To rigorously investigate this, we propose a controlled data processing pipeline based on video composition that results in TRACE-Edit, a diagnostic dataset focusing on relation-based editing. Leveraging this dataset, we propose a comprehensive diagnostic protocol to analyze two important designs of meta-query and connector in the existing video editing models. Systematic evaluation of four representative model cases reveals that fine-grained structural semantics can be severely degraded during alignment. Our findings overturn the assumption of lossless semantic transfer, identifying the VLM-to-DiT alignment as a major bottleneck and providing a new diagnostic foundation for future multi-modal alignment architectures.

Problem

• VLM-to-DiT alignment
• semantic bottleneck
• video editing
• structural semantics
• multi-modal alignment

Innovation

• VLM-to-DiT alignment
• semantic bottleneck
• TRACE-Edit
• relation-based editing
• diagnostic protocol