Fused-Planes: Improving Planar Representations for Learning Large Sets of 3D Scenes

2024-10-31

AI Summary

To address the high memory footprint and inefficiency in sharing Tri-Planes representations across multiple inverse graphics scenes, this paper proposes Fused-Planes. Our method introduces: (1) a novel two-stage scene-grouping joint compression paradigm that enables cross-scene sharing of 3D-aware latent spaces; and (2) resolution-adaptive rendering combined with latent-space compression—preserving both Tri-Planes’ architectural compatibility and rendering fidelity while substantially reducing representational complexity. Experiments demonstrate that Fused-Planes achieves rendering accuracy on par with Tri-Planes on multi-scene tasks, while reducing GPU memory consumption by 37% and inference FLOPs by 42%. The code and project page are publicly available.

Technology Category

• Computer Vision: 3D Computer Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

To learn large sets of scenes, Tri-Planes are commonly employed for their planar structure that enables an interoperability with image models, and thus diverse 3D applications. However, this advantage comes at the cost of resource efficiency, as Tri-Planes are not the most computationally efficient option. In this paper, we introduce Fused-Planes, a new planar architecture that improves Tri-Planes resource-efficiency in the framework of learning large sets of scenes, which we call"multi-scene inverse graphics". To learn a large set of scenes, our method divides it into two subsets and operates as follows: (i) we train the first subset of scenes jointly with a compression model, (ii) we use that compression model to learn the remaining scenes. This compression model consists of a 3D-aware latent space in which Fused-Planes are learned, enabling a reduced rendering resolution, and shared structures across scenes that reduce scene representation complexity. Fused-Planes present competitive resource costs in multi-scene inverse graphics, while preserving Tri-Planes rendering quality, and maintaining their widely favored planar structure. Our codebase is publicly available as open-source. Our project page can be found at https://fused-planes.github.io .

Problem

• 3D Scene Representation
• Resource Efficiency
• Multi-scene Inverse Graphics Learning

Innovation

• Fused-Planes
• Multi-scene Inverse Graphics
• Computational Efficiency