Unleashing Vecset Diffusion Model for Fast Shape Generation

2025-03-20

AI Summary

To address the slow inference speed of Vecset diffusion models (VDMs) in high-resolution 3D shape generation, this work proposes a dual-path acceleration framework. First, Progressive Flow Distillation introduces a five-step efficient DiT sampling strategy based on stable consistency distillation. Second, the Lightning Vecset Decoder integrates adaptive KV compression, hierarchical voxel decoding, and a lightweight network architecture. Built upon Hunyuan3D-2, our Turbo variant achieves end-to-end acceleration for both VAE decoding and diffusion sampling while preserving state-of-the-art reconstruction and generation quality—yielding 45× and 32× speedups in reconstruction and generation inference, respectively. The core contribution lies in the first synergistic optimization of consistency distillation with Vecset’s sparse voxel representation, significantly overcoming the real-time generation bottleneck in 3D diffusion modeling.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization

Abstract

3D shape generation has greatly flourished through the development of so-called"native"3D diffusion, particularly through the Vecset Diffusion Model (VDM). While recent advancements have shown promising results in generating high-resolution 3D shapes, VDM still struggles with high-speed generation. Challenges exist because of difficulties not only in accelerating diffusion sampling but also VAE decoding in VDM, areas under-explored in previous works. To address these challenges, we present FlashVDM, a systematic framework for accelerating both VAE and DiT in VDM. For DiT, FlashVDM enables flexible diffusion sampling with as few as 5 inference steps and comparable quality, which is made possible by stabilizing consistency distillation with our newly introduced Progressive Flow Distillation. For VAE, we introduce a lightning vecset decoder equipped with Adaptive KV Selection, Hierarchical Volume Decoding, and Efficient Network Design. By exploiting the locality of the vecset and the sparsity of shape surface in the volume, our decoder drastically lowers FLOPs, minimizing the overall decoding overhead. We apply FlashVDM to Hunyuan3D-2 to obtain Hunyuan3D-2 Turbo. Through systematic evaluation, we show that our model significantly outperforms existing fast 3D generation methods, achieving comparable performance to the state-of-the-art while reducing inference time by over 45x for reconstruction and 32x for generation. Code and models are available at https://github.com/Tencent/FlashVDM.

Problem

• Accelerates 3D shape generation using Vecset Diffusion Model
• Reduces VAE decoding overhead with efficient network design
• Improves diffusion sampling speed with fewer inference steps

Innovation

• Progressive Flow Distillation stabilizes consistency distillation
• Adaptive KV Selection enhances VAE decoding efficiency
• Hierarchical Volume Decoding reduces FLOPs significantly