FB-4D: Spatial-Temporal Coherent Dynamic 3D Content Generation with Feature Banks

2025-03-26

AI Summary

High-fidelity 4D (dynamic 3D) content generation suffers from weak spatiotemporal consistency. To address this, we propose a generative framework centered on an updateable Feature Bank, which enables cross-frame feature storage, dynamic fusion, and multi-view spatiotemporal alignment for robust, temporally coherent dynamic 3D reconstruction. Our key contributions are threefold: (1) the first introduction of an online-updatable Feature Bank mechanism; (2) a novel multi-round autoregressive reference sequence generation paradigm that iteratively refines output quality; and (3) zero-shot performance competitive with fully supervised trained methods—without any fine-tuning. Experiments demonstrate significant improvements in rendering fidelity, spatiotemporal consistency, and robustness. Our approach outperforms existing zero-shot multi-view generation methods across all metrics and achieves performance on par with supervised training-based approaches.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

With the rapid advancements in diffusion models and 3D generation techniques, dynamic 3D content generation has become a crucial research area. However, achieving high-fidelity 4D (dynamic 3D) generation with strong spatial-temporal consistency remains a challenging task. Inspired by recent findings that pretrained diffusion features capture rich correspondences, we propose FB-4D, a novel 4D generation framework that integrates a Feature Bank mechanism to enhance both spatial and temporal consistency in generated frames. In FB-4D, we store features extracted from previous frames and fuse them into the process of generating subsequent frames, ensuring consistent characteristics across both time and multiple views. To ensure a compact representation, the Feature Bank is updated by a proposed dynamic merging mechanism. Leveraging this Feature Bank, we demonstrate for the first time that generating additional reference sequences through multiple autoregressive iterations can continuously improve generation performance. Experimental results show that FB-4D significantly outperforms existing methods in terms of rendering quality, spatial-temporal consistency, and robustness. It surpasses all multi-view generation tuning-free approaches by a large margin and achieves performance on par with training-based methods.

Problem

• Achieving high-fidelity 4D generation with spatial-temporal consistency
• Enhancing consistency in dynamic 3D content across frames
• Improving rendering quality and robustness in 4D generation

Innovation

• Feature Bank mechanism enhances spatial-temporal consistency
• Dynamic merging ensures compact feature representation
• Autoregressive iterations improve generation performance continuously