fVDB : A Deep-Learning Framework for Sparse, Large Scale, and High Performance Spatial Intelligence

2024-07-01

ACM Transactions on Graphics

AI Summary

To address memory explosion and computational inefficiency in deep learning with large-scale sparse 3D data (e.g., high-resolution point clouds, unbounded NeRF), this paper introduces fVDB—the first GPU-optimized framework for end-to-end differentiable 3D spatial intelligence. Methodologically, fVDB features: (1) a dynamic sparse voxel grid representation built upon a novel VDB (Volume Data Base) indexing scheme; (2) CUDA/Tensor Core-accelerated sparse operations—including convolution, attention, differentiable ray tracing (via the HDDA algorithm), and mesh generation; and (3) native PyTorch integration with support for jagged tensors. Experiments demonstrate that fVDB achieves 5–12% higher accuracy, 3–8× higher throughput, and 40–70% lower memory consumption compared to state-of-the-art methods on point cloud segmentation, 3D generation, unbounded NeRF rendering, and large-scale reconstruction. Moreover, it enables up to 100× scaling in spatial resolution and dataset size.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Data Mining & Knowledge Management: Mining of Spatial, Temporal or Spatio-Temporal Data

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

We present fVDB, a novel GPU-optimized framework for deep learning on large-scale 3D data. fVDB provides a complete set of differentiable primitives to build deep learning architectures for common tasks in 3D learning such as convolution, pooling, attention, ray-tracing, meshing, etc. fVDB simultaneously provides a much larger feature set (primitives and operators) than established frameworks with no loss in efficiency: our operators match or exceed the performance of other frameworks with narrower scope. Furthermore, fVDB can process datasets with much larger footprint and spatial resolution than prior works, while providing a competitive memory footprint on small inputs. To achieve this combination of versatility and performance, fVDB relies on a single novel VDB index grid acceleration structure paired with several key innovations including GPU accelerated sparse grid construction, convolution using tensorcores, fast ray tracing kernels using a Hierarchical Digital Differential Analyzer algorithm (HDDA), and jagged tensors. Our framework is fully integrated with PyTorch enabling interoperability with existing pipelines, and we demonstrate its effectiveness on a number of representative tasks such as large-scale point-cloud segmentation, high resolution 3D generative modeling, unbounded scale Neural Radiance Fields, and large-scale point cloud reconstruction.

Problem

• Developing GPU-optimized framework for large-scale 3D deep learning
• Providing versatile differentiable operators for 3D tasks efficiently
• Enabling high-performance processing of sparse, high-resolution spatial data

Innovation

• GPU-optimized framework for large-scale 3D data
• Single VDB index grid with multiple acceleration innovations
• Fully integrated with PyTorch for existing pipelines