Hyper-3DG: Text-to-3D Gaussian Generation via Hypergraph

2024-03-14

International Journal of Computer Vision

AI Summary

In text-to-3D generation, insufficient high-order coupling modeling between geometry and texture leads to structural and textural artifacts—including oversmoothing, oversaturation, and Janus-like inconsistencies. To address this, we propose an end-to-end, text-driven framework for 3D Gaussian model generation. Its core innovation is the Geometry and Texture Hypergraph Refiner (HGRefiner), the first module to incorporate hypergraph learning into 3D Gaussian optimization. HGRefiner performs patch-level hypergraph modeling and dual-stream feature refinement, enabling joint updating of explicit geometric/texture attributes and implicit features. Built upon 3D Gaussian Splatting, our method incurs zero additional inference overhead. On standard benchmarks, it achieves state-of-the-art performance across PSNR, SSIM, and LPIPS—demonstrating significant suppression of structural distortions and textural artifacts.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Text-to-3D generation represents an exciting field that has seen rapid advancements, facilitating the transformation of textual descriptions into detailed 3D models. However, current progress often neglects the intricate high-order correlation of geometry and texture within 3D objects, leading to challenges such as over-smoothness, over-saturation and the Janus problem. In this work, we propose a method named ``3D Gaussian Generation via Hypergraph (Hyper-3DG)'', designed to capture the sophisticated high-order correlations present within 3D objects. Our framework is anchored by a well-established mainflow and an essential module, named ``Geometry and Texture Hypergraph Refiner (HGRefiner)''. This module not only refines the representation of 3D Gaussians but also accelerates the update process of these 3D Gaussians by conducting the Patch-3DGS Hypergraph Learning on both explicit attributes and latent visual features. Our framework allows for the production of finely generated 3D objects within a cohesive optimization, effectively circumventing degradation. Extensive experimentation has shown that our proposed method significantly enhances the quality of 3D generation while incurring no additional computational overhead for the underlying framework. (Project code: https://github.com/yjhboy/Hyper3DG)

Problem

• Text-to-3D generation with high-order geometry and texture correlation
• Overcomes over-smoothness, over-saturation, and Janus problem in 3D models
• Enhances 3D Gaussian representation and accelerates update process

Innovation

• Hypergraph for 3D generation
• Geometry and Texture Hypergraph Refiner
• Patch-3DGS Hypergraph Learning