Feed-forward Motion In-betweening for Any 4D

2026-06-20

AI Summary

Existing 4D generation methods struggle with slow inference, limited spatiotemporal controllability, and error accumulation when synthesizing long-duration animations of meshes with arbitrary topologies. This work proposes a feedforward, keyframe-conditioned framework for 4D intermediate frame generation, which constructs a topology-agnostic latent space via a per-frame mesh VAE and integrates a keyframe anchoring mechanism with an MMDiT-based rectified flow model to enable efficient and highly controllable motion interpolation. Evaluated on the DyMesh16 and DyMesh32 benchmarks, the method significantly outperforms current approaches, achieving high-fidelity results while effectively mitigating error propagation in long sequences and substantially improving both spatiotemporal controllability and inference efficiency.

Abstract

4D dynamics (3D geometry evolving over time) is a fundamental representation of the physical world and plays a crucial role in world modeling (e.g., animation and games). Owing to the scarcity of large-scale, long-horizon 4D mesh data with arbitrary shapes, early text-to-4D methods rely on distillation or test-time optimization from video diffusion priors, making inference prohibitively slow. Recent feed-forward generators greatly reduce inference cost but offer limited spatiotemporal controllability, and short-horizon generation often leads to error accumulation in long-horizon sequences. We propose a novel feed-forward in-betweening framework for arbitrary 4D meshes with keyframe conditioning. Building on universal mesh-animation latents, we introduce a frame-wise mesh VAE that encodes each frame into topology-agnostic latent tokens anchored by a reference mesh for keyframe conditioning. We further introduce a keyframe-conditioned rectified flow model with an MMDiT backbone that synthesizes non-keyframe frames conditioned on sparse keyframes. Experiments show strong performance and improved controllability on both DyMesh16 and DyMesh32 benchmarks.

Problem

• 4D generation
• mesh animation
• spatiotemporal controllability
• long-horizon sequence
• keyframe conditioning

Innovation

• feed-forward in-betweening
• 4D mesh generation
• keyframe conditioning
• mesh VAE
• rectified flow