What If : Understanding Motion Through Sparse Interactions

2025-10-14

AI Summary

This work addresses the challenge of multimodal dynamic modeling in physical scenes induced by sparse, localized interactions (“pokes”). Existing approaches fail to explicitly capture the multimodality, uncertainty, and causal dependence of motion on such interactions. To this end, we propose the Flow Poke Transformer (FPT), the first framework capable of directly predicting explicit motion distributions under sparse pokes. FPT integrates optical flow priors with a Transformer architecture and employs self-supervised learning to model the coupling between local interactions and global motion dynamics, yielding interpretable, probabilistic motion representations. We evaluate FPT on three distinct tasks—facial motion generation, articulated object motion estimation, and motion-part segmentation—where it consistently outperforms state-of-the-art methods. Moreover, FPT demonstrates strong cross-domain generalization and holds promise for unified downstream task processing.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Multimodal Learning
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Understanding the dynamics of a physical scene involves reasoning about the diverse ways it can potentially change, especially as a result of local interactions. We present the Flow Poke Transformer (FPT), a novel framework for directly predicting the distribution of local motion, conditioned on sparse interactions termed"pokes". Unlike traditional methods that typically only enable dense sampling of a single realization of scene dynamics, FPT provides an interpretable directly accessible representation of multi-modal scene motion, its dependency on physical interactions and the inherent uncertainties of scene dynamics. We also evaluate our model on several downstream tasks to enable comparisons with prior methods and highlight the flexibility of our approach. On dense face motion generation, our generic pre-trained model surpasses specialized baselines. FPT can be fine-tuned in strongly out-of-distribution tasks such as synthetic datasets to enable significant improvements over in-domain methods in articulated object motion estimation. Additionally, predicting explicit motion distributions directly enables our method to achieve competitive performance on tasks like moving part segmentation from pokes which further demonstrates the versatility of our FPT. Code and models are publicly available at https://compvis.github.io/flow-poke-transformer.

Problem

• Predicting multi-modal motion distributions from sparse interactions
• Modeling scene dynamics dependencies on physical interactions
• Estimating inherent uncertainties in physical scene dynamics

Innovation

• Flow Poke Transformer predicts motion from sparse pokes
• It provides interpretable multi-modal motion distributions
• Model enables competitive performance across diverse tasks