What Moves? Localized Motion Representations for Compositional Scene Control

2026-09-03

AI Summary

该研究解决了视频中局部运动表示问题，通过全视频处理和基于空间掩码的区域查询方法，生成保留全局上下文的局部动态嵌入。

Abstract

Real-world dynamics are inherently compositional: multiple entities move simultaneously within a shared scene, each exhibiting distinct motion patterns. Yet most existing video representations encode motion globally, without explicitly capturing localized motion for individual entities. Crucially, motion is defined relative to a global reference frame, including camera motion and scene layout. However, localized embeddings are often computed from cropped images or obtained by masking features after encoding, discarding the context needed to interpret motion. To address this, we introduce a promptable localized motion representation that produces persistent embeddings for user-specified regions defined by spatial masks. Rather than cropping the input or masking features, our model processes the full video and conditions motion encoding directly on the queried region. This yields temporally consistent, region-addressable embeddings that isolate local dynamics while retaining the global context required for disambiguation. We demonstrate object-level motion transfer, enabling controlled composition of dynamic scenes. Beyond generative control, our embeddings support localized action classification in multi-actor videos. Across both tasks, our approach improves controllability and outperforms global representations localized through cropping or post-hoc masking. Project Page: https://compvis.github.io/WhatMoves

Problem

• Localized Motion
• Compositional Scene Control
• Global Reference Frame
• Persistent Embeddings
• Multi-actor Videos

Innovation

• localized motion representation
• persistent embeddings
• spatial masks
• full video processing
• region-addressable