TriMotion: Modality-Agnostic Camera Control for Video Generation

2026-06-18

AI Summary

This work addresses the challenge of flexibly supporting multimodal camera motion control in video generation. To this end, the authors propose a modality-agnostic framework that maps video, pose, and text inputs into a unified motion embedding space, enabling consistent and precise viewpoint manipulation. Key contributions include the construction of a Motion Triplet Dataset, the introduction of a geometry-driven motion representation based on camera extrinsics, and the design of a motion consistency objective in the latent space. The proposed method not only unifies multimodal inputs under a single processing pipeline but also enables novel capabilities such as motion sequence composition and cross-modal interpolation. Experiments demonstrate that the approach generates high-quality videos across all three modalities, accurately adhering to target camera trajectories and validating its effectiveness and generalization.

Abstract

Camera motion control is essential for directing viewpoint changes in generative systems. However, existing methods typically condition the generation process on a single specific modality, such as explicit pose trajectories or reference videos, limiting their ability to support heterogeneous user inputs. To address this limitation, we present TriMotion, a modality-agnostic framework for camera-controlled video generation that maps video, pose, and text inputs, describing the same camera trajectory into a shared motion embedding space. Learning such a space requires synchronized supervision across modalities. Therefore, we build the Motion Triplet Dataset by extending a Multi-Cam Video Dataset with geometry-grounded motion descriptions derived from camera extrinsics. We further introduce a latent motion consistency objective that leverages the motion embedding space to encourage the generated video to follow the target camera trajectory directly in latent space, avoiding the cost of pixel-space decoding. Extensive experiments show that TriMotion generates high-quality videos that accurately follow the target camera trajectories across all three modalities. Beyond standard generation, the shared motion embedding space also enables flexible applications such as sequential motion composition and cross-modal motion interpolation.

Problem

• camera motion control
• modality-agnostic
• video generation
• heterogeneous inputs
• motion embedding

Innovation

• modality-agnostic
• camera motion control
• shared motion embedding
• latent motion consistency
• cross-modal interpolation