Rethinking Image-to-3D Generation with Sparse Queries: Efficiency, Capacity, and Input-View Bias

2026-04-15

AI Summary

Existing image-to-3D generation methods often suffer from high computational costs, substantial memory consumption, and strong dependence on input viewpoints. This work proposes SparseGen, a framework that introduces sparsely learnable 3D anchor queries coupled with a local Gaussian primitive expansion mechanism to enable efficient 3D generation without requiring explicit 3D supervision. By leveraging rectified-flow reconstruction objectives and a sparse implicit representation, SparseGen significantly mitigates view bias while enhancing representational efficiency and generalization. Experimental results demonstrate that SparseGen achieves strong multi-view consistency, substantially reduces memory usage and inference time, and exhibits lower sensitivity to input viewpoints alongside higher representational utilization, as validated by quantitative metrics.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We present SparseGen, a novel framework for efficient image-to-3D generation, which exhibits low input-view bias while being significantly faster. Unlike traditional approaches that rely on dense volumetric grids, triplanes, or pixel-aligned primitives, we model scenes with a compact sparse set of learned 3D anchor queries and a learned expansion operator that decodes each transformed query into a small local set of 3D Gaussian primitives. Trained under a rectified-flow reconstruction objective without 3D supervision, our model learns to allocate representation capacity where geometry and appearance matter, achieving significant reductions in memory and inference time while preserving multi-view fidelity. We introduce quantitative measures of input-view bias and utilization to show that sparse queries reduce overfitting to conditioning views while being representationally efficient. Our results argue that sparse set-latent expansion is a principled, practical alternative for efficient 3D generative modeling.

Problem

• image-to-3D generation
• input-view bias
• efficiency
• representation capacity
• sparse representation

Innovation

• sparse queries
• 3D Gaussian primitives
• rectified-flow
• input-view bias
• efficient 3D generation