Alignment Is All You Need For X-to-4D Generation

2026-07-02

AI Summary

This work addresses the challenges in arbitrary-modality-to-4D (X-to-4D) generation, which are primarily hindered by the scarcity of high-quality, diverse data and the limited scalability of existing approaches. To overcome these limitations, we propose Align4D, a unified framework that maps inputs from any modality into video–3D pairs, leveraging videos to guide 4D dynamics and 3D representations to shape geometric structure. Key innovations include object distance alignment (VAOD/MAOD), joint motion–geometry constraints, and decoupled asynchronous optimization of Gaussian attributes and deformation networks. We also introduce X4D, the first benchmark dataset supporting multimodal inputs for 4D generation. Extensive experiments demonstrate that Align4D achieves state-of-the-art performance on both the X4D and Consistent4D datasets, significantly improving generation quality and spatiotemporal consistency.

Abstract

Generative diffusion models excel at synthesizing high-quality images, videos, and 3D content under multimodal control. However, arbitrary user-defined modality-to-4D (X-to-4D) generation remains challenging due to the high cost of constructing diverse datasets and the limited scalability of existing methods. This paper presents Align4D, a flexible framework that translates any-modal input into coherent video-3D pairs, using video to guide 4D motion and 3D data to shape 4D geometry. Align4D introduces three key techniques: (1) Object Distance Alignment, which searches Video-Aligned and Multiview-Aligned Object Distances (VAOD/MAOD), respectively, to reconcile 4D renderings with video and the priors of multiview diffusion models; (2) Motion-Geometry Joint Alignment, which constrains known and unknown views through synchronized video and 3D inputs, ensuring consistent 4D generation; and (3) Asynchronous Optimization, which decouples Gaussian attribute and deformation network training to enhance motion and geometry fidelity. We further propose the X4D dataset, which integrates prompt, image, video, and 3D data for benchmarking. Experiments on X4D and Consistent4D demonstrate that Align4D achieves state-of-the-art quality and consistency in X-to-4D generation. Project page: https://miaoqiaowei.github.io/Align4D/.

Problem

• X-to-4D generation
• 4D content synthesis
• multimodal control
• scalability
• dataset diversity

Innovation

• X-to-4D generation
• Object Distance Alignment
• Motion-Geometry Joint Alignment
• Asynchronous Optimization
• Align4D