CoGen3D: An Agentic Human-AI Co-Design Pipeline for 3D Asset Generation for Virtual Reality

2026-07-04

AI Summary

This study addresses the high barrier to entry in 3D asset creation with current generative AI tools, which lack conversational mechanisms to guide users in articulating design intent and validating proposals. To overcome this limitation, we propose an intent-driven, agent-mediated human-AI co-design paradigm that structures collaboration into distinct phases: eliciting user intent through dialogue, confirming concept images, and generating 3D assets directly deployable in virtual reality (VR) environments. Our end-to-end pipeline integrates conversational AI, concept image generation, and image-to-3D techniques, shifting VR content creation from technical execution toward collaborative spatial design. A user study (N=120) demonstrates that this approach significantly enhances scene engagement and alters emotional responses; participants consistently preferred co-designed concept images and exhibited no self-authorship bias toward their own creations.

Abstract

Creating 3D assets for virtual reality requires modeling expertise, which restricts the authorship of immersive experiences. Existing generative AI tools rely on unconstrained, command-driven prompting, lacking the conversational scaffolding needed for users to articulate their intent and validate designs prior to rendering. To address this, we introduce CoGen3D, an agentic human-AI co-design pipeline that proactively guides users through conversational intent elicitation, a concept image confirmation, and image-to-3D generation that directly deploys to immersive scenes. We evaluated this system through a user study (N=120) across six affectively diverse immersive scenes, observing 60 Design group participants who co-created 3D assets for the scenes, and 60 Validation group participants who experienced the scenes with generated assets. Our findings show that co-designed assets are associated with higher scene engagement and shifted affective responses, while participants generally preferred concept images over the final 3D assets, with no increased leniency toward degradation in their own creations. Analysis of the human-AI conversations further shows that target environments shape users' conversational patterns. Our results suggest that our staged, intent-based co-design can democratize virtual reality authoring and shift immersive content creation from technical execution toward collaborative spatial design.

Problem

• 3D asset generation
• virtual reality
• human-AI co-design
• generative AI
• intent elicitation

Innovation

• agentic co-design
• conversational intent elicitation
• image-to-3D generation
• virtual reality authoring
• human-AI collaboration