🤖 AI Summary
Existing 3D novel view synthesis (NVS) methods rely heavily on dense 3D annotations or multi-view inputs, limiting generalization; conversely, 3D-free approaches enable text-driven generation of complex scenes but lack precise camera control. This paper proposes a single-image-driven, camera-controllable NVS framework. For the first time, it incorporates a pre-trained NVS model as a weak geometric prior into a 3D-free diffusion architecture and explicitly encodes camera pose in the CLIP embedding space. Built upon Stable Diffusion, our method integrates cross-modal CLIP enhancement, knowledge distillation, and pose-conditioned embedding—requiring neither multi-view nor 3D supervision. Evaluated on diverse real-world scenes, it significantly outperforms state-of-the-art methods, enabling accurate, arbitrary-view synthesis with high fidelity, geometric consistency, and fine-grained detail. Both qualitative and quantitative evaluations demonstrate leading performance.
📝 Abstract
Recent 3D novel view synthesis (NVS) methods often require extensive 3D data for training, and also typically lack generalization beyond the training distribution. Moreover, they tend to be object centric and struggle with complex and intricate scenes. Conversely, 3D-free methods can generate text-controlled views of complex, in-the-wild scenes using a pretrained stable diffusion model without the need for a large amount of 3D-based training data, but lack camera control. In this paper, we introduce a method capable of generating camera-controlled viewpoints from a single input image, by combining the benefits of 3D-free and 3D-based approaches. Our method excels in handling complex and diverse scenes without extensive training or additional 3D and multiview data. It leverages widely available pretrained NVS models for weak guidance, integrating this knowledge into a 3D-free view synthesis style approach, along with enriching the CLIP vision-language space with 3D camera angle information, to achieve the desired results. Experimental results demonstrate that our method outperforms existing models in both qualitative and quantitative evaluations, achieving high-fidelity, consistent novel view synthesis at desired camera angles across a wide variety of scenes while maintaining accurate, natural detail representation and image clarity across various viewpoints. We also support our method with a comprehensive analysis of 2D image generation models and the 3D space, providing a solid foundation and rationale for our solution.