MATRIX: Mask Track Alignment for Interaction-aware Video Generation

2025-10-08

AI Summary

Video diffusion Transformers (Video DiTs) struggle to model multi-instance scenarios and dynamic subject-object interactions, with their internal interaction representation mechanisms remaining poorly understood. To address this, we propose MATRIX: an interaction-aware masked trajectory alignment regularization framework that, for the first time, systematically models semantic binding and cross-frame propagation within critical interaction layers of Video DiTs. MATRIX is trained end-to-end on the MATRIX-11K dataset by jointly leveraging video-to-text and video-to-video attention analysis alongside masked trajectory supervision. Experiments demonstrate substantial improvements in interaction fidelity and semantic alignment accuracy of generated videos, effectively mitigating object drift and hallucination. Comprehensive evaluation under the InterGenEval benchmark confirms the method’s efficacy, establishing a new state-of-the-art for interaction-aware video generation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Matrix & Tensor Methods
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Video DiTs have advanced video generation, yet they still struggle to model multi-instance or subject-object interactions. This raises a key question: How do these models internally represent interactions? To answer this, we curate MATRIX-11K, a video dataset with interaction-aware captions and multi-instance mask tracks. Using this dataset, we conduct a systematic analysis that formalizes two perspectives of video DiTs: semantic grounding, via video-to-text attention, which evaluates whether noun and verb tokens capture instances and their relations; and semantic propagation, via video-to-video attention, which assesses whether instance bindings persist across frames. We find both effects concentrate in a small subset of interaction-dominant layers. Motivated by this, we introduce MATRIX, a simple and effective regularization that aligns attention in specific layers of video DiTs with multi-instance mask tracks from the MATRIX-11K dataset, enhancing both grounding and propagation. We further propose InterGenEval, an evaluation protocol for interaction-aware video generation. In experiments, MATRIX improves both interaction fidelity and semantic alignment while reducing drift and hallucination. Extensive ablations validate our design choices. Codes and weights will be released.

Problem

• Video DiTs struggle with modeling multi-instance and subject-object interactions
• Current models lack internal representations for interaction dynamics in videos
• Need better evaluation protocols for interaction-aware video generation

Innovation

• Aligns attention layers with multi-instance mask tracks
• Introduces regularization for interaction-dominant video DiT layers
• Proposes InterGenEval protocol for interaction-aware video evaluation