Controllable 3D Object Generation with Single Image Prompt

2025-11-27

AI Summary

Existing text-to-3D generation methods rely on costly text inversion or pseudo-embedding optimization, suffering from high training overhead and coarse-grained control. This paper proposes a training-free, single-image-guided 3D generation framework. We design a plug-and-play image adapter that directly maps an input image to controllable diffusion model conditions, bypassing textual priors entirely. Additionally, we introduce a depth-conditioned warm-up mechanism that injects geometric priors during early denoising steps, significantly improving 3D consistency and enabling precise control over pose and depth. Our approach completely eliminates text inversion and embedding optimization, supporting fine-grained joint control via text, depth maps, and camera viewpoints. Quantitative and qualitative evaluations demonstrate superior image fidelity and geometric consistency compared to state-of-the-art baselines. A user study further confirms the effectiveness of our control paradigm and the high perceptual quality of generated outputs.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recently, the impressive generative capabilities of diffusion models have been demonstrated, producing images with remarkable fidelity. Particularly, existing methods for the 3D object generation tasks, which is one of the fastest-growing segments in computer vision, pre-dominantly use text-to-image diffusion models with textual inversion which train a pseudo text prompt to describe the given image. In practice, various text-to-image generative models employ textual inversion to learn concepts or styles of target object in the pseudo text prompt embedding space, thereby generating sophisticated outputs. However, textual inversion requires additional training time and lacks control ability. To tackle this issues, we propose two innovative methods: (1) using an off-the-shelf image adapter that generates 3D objects without textual inversion, offering enhanced control over conditions such as depth, pose, and text. (2) a depth conditioned warmup strategy to enhance 3D consistency. In experimental results, ours show qualitatively and quantitatively comparable performance and improved 3D consistency to the existing text-inversion-based alternatives. Furthermore, we conduct a user study to assess (i) how well results match the input image and (ii) whether 3D consistency is maintained. User study results show that our model outperforms the alternatives, validating the effectiveness of our approaches. Our code is available at GitHub repository:https://github.com/Seooooooogi/Control3D_IP/

Problem

• Generates 3D objects from single images without textual inversion.
• Enhances control over depth, pose, and text conditions.
• Improves 3D consistency compared to text-inversion methods.

Innovation

• Uses image adapter without textual inversion
• Implements depth conditioned warmup strategy
• Enhances 3D consistency and control