Fast 4D Mesh Generation by Spatio-Temporal Attention Chains

2026-05-19

AI Summary

Existing 4D mesh generation methods suffer from high computational costs, poor scalability to long videos, and insufficient temporal consistency. This work proposes a training-free, efficient framework that reveals, for the first time, reliable temporal correspondences already present in the early layers of 4D backbone networks. Leveraging this insight, the authors design a spatiotemporal attention chain that propagates spatiotemporal information from anchor grids in latent space, enabling high-quality dynamic mesh generation without explicit matching. The method supports novel applications such as camera estimation and achieves state-of-the-art performance in both 2D and 4D zero-shot tracking. With an inference speed of 9 seconds per sequence—13× faster than the current best approach—it can process video sequences up to 16× longer.

Abstract

4D mesh generation has recently emerged as a powerful paradigm for recovering dynamic 3D structure from videos, but existing methods remain slow, computationally expensive, and difficult to scale to longer sequences. We introduce a training-free approach that accelerates 4D mesh generation while improving temporal correspondence quality. Our key observation is that temporal correspondences emerge inside a 4D backbone long before its generated meshes become visually accurate. We exploit this with a general framework we call Spatio-Temporal Attention Chain which propagates information across space and time. Starting from vertices on an anchor mesh, the chain maps vertices to latent tokens. It then follows temporal correspondences in latent space, and recovers frame-specific vertices through latent-to-vertex attention. This design avoids expensive explicit matching while preserving anchor mesh details and thereby improving dynamic mesh geometry and temporal consistency. Compared to state-of-the-art, our method generates a 4D mesh in 9 seconds, achieving a $13\times$ speedup while producing higher-quality results. Moreover, our approach scales to videos up to $16\times$ longer without degrading mesh quality. Beyond generation, the improved correspondences enable competitive zero-shot performance on two downstream tasks: 2D object tracking and 4D tracking. We further show that our framework enables reliable camera estimation, a capability not supported by prior 4D mesh generation methods.

Problem

• 4D mesh generation
• temporal correspondence
• computational efficiency
• scalability
• dynamic 3D structure

Innovation

• 4D mesh generation
• spatio-temporal attention
• temporal correspondence
• training-free
• latent-to-vertex attention