CoMoGen: COntrollable MOtion Dynamics and Interactions with Mask-Guided Video GENeration

2026-05-21

AI Summary

This work proposes an efficient and controllable video generation method that drives subject motion using a single image and a binary mask sequence while preserving realistic environmental interactions. The key innovation lies in identifying critical “motion layers” within a unified MMDiT architecture and applying LoRA fine-tuning exclusively to these layers. Additionally, a lightweight MaskAdapter is introduced to encode the mask sequence into latent residual signals, which are injected into the model via cosine-weighted scheduling. Without modifying the backbone network, the approach achieves precise control over dynamic content and establishes state-of-the-art performance in motion fidelity and perceptual realism across multiple datasets, significantly outperforming existing methods with lower computational overhead.

Abstract

We present CoMoGen, a controllable video generation framework that generates realistic interactive dynamics from a single binary mask sequence conditioned on an input image. CoMoGen introduces a lightweight MaskAdapter that encodes binary mask sequences into a latent residual signal, injected into the Multi Modal Diffusion Transformer (MMDiT) model through a cosine-weighted schedule. Unlike the hierarchical coarse-to-fine design of UNet architectures, MMDiT operates as a sequence of uniform transformer blocks, making it difficult to identify which layers are responsible for the motion generation. Therefore, we propose a novel way to determine "Motion Layers" operating in the attention space of MMDiT. We fine-tune the model by using Low-Rank Adaptation (LoRA) to the Motion Layers, without requiring any architecture change in the MMDiT. This selective adaptation enables our method to focus on motion-critical components, yielding reduced computational cost. Despite its simplicity, CoMoGen enables precise subject motion and plausible interactions with surrounding humans, objects, and scenes. Comprehensive experiments on different datasets show that CoMoGen consistently outperforms prior controllable video generation methods and achieves state-of-the-art performance in motion fidelity and perceptual realism. Project page: mericadil.github.io/CoMoGen.

Problem

• controllable video generation
• motion dynamics
• mask-guided generation
• interactive video synthesis
• motion fidelity

Innovation

• MaskAdapter
• Motion Layers
• MMDiT
• LoRA
• controllable video generation