MiXR: Harvesting and Recomposing Geometry from Real-World Objects for In-Situ 3D Design

2026-05-10

AI Summary

Existing 3D generative AI systems struggle to precisely control spatial structure, limiting their utility in complex geometric design. This work proposes a hybrid modeling framework that integrates extended reality (XR), environmental geometry capture, interactive 3D manipulation, and generative AI. Users extract object fragments from real-world scenes and explicitly define structural arrangements through intuitive spatial composition; the system then leverages generative AI to synthesize geometrically coherent, detailed full 3D models. By grounding spatial intent in tangible user-defined structures rather than relying solely on text or image prompts, the approach overcomes key limitations in expressing complex spatial relationships. User studies demonstrate that, compared to purely generative baselines, the proposed system significantly improves design accuracy and user sense of control while reducing cognitive load.

Abstract

Recent developments in 3D generative AI enable users to create bespoke 3D models from text or image prompts. However, these approaches provide limited control over spatial structure, making them ill suited for tasks requiring precise geometric composition. We present MiXR, an XR system for in-situ compositional modeling that enables users to create new 3D models by harvesting geometry from their environment. Users extract segments from captured objects and assemble new artifacts through direct 3D manipulation, while generative AI synthesizes a coherent model from the user-defined composition. This hybrid workflow allows users to define spatial structure explicitly while delegating geometric refinement to generative models, enabling them to specify spatial intent that is difficult to express through verbal prompts alone. In a controlled user study ($N=12$), participants using MiXR rated their designs as significantly closer to the target, felt more in control, and experienced lower cognitive workload compared to a generative composition baseline.

Problem

• 3D generative AI
• spatial structure
• geometric composition
• in-situ modeling

Innovation

• in-situ 3D design
• geometry harvesting
• generative AI
• spatial composition
• XR system