CRAFT: Designing Creative and Functional 3D Objects

2024-12-05

AI Summary

Existing AI-driven design tools struggle to simultaneously satisfy anatomical constraints and semantic design intent, limiting the functional ergonomics of everyday objects. This paper introduces the first differentiable mesh deformation framework that unifies semantic alignment, contact relationship modeling, and geometric interpenetration constraints—enabling human-aware, text/image/sketch-driven 3D generation. Our method integrates CLIP- and diffusion-based semantic guidance with physics-informed contact modeling and a penetration-aware loss function, facilitating end-to-end unsupervised optimization. Evaluated across diverse household items, our approach yields qualitatively natural human–object interaction postures and quantitatively improves contact accuracy by 23.6% over baselines. It also significantly enhances semantic fidelity and anatomical fit precision, demonstrating superior capability in generating functionally appropriate, user-centered designs.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Computer Vision: Diffusion Models for Vision
• Intelligent Robots: Manipulation

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

For designing a wide range of everyday objects, the design process should be aware of both the human body and the underlying semantics of the design specification. However, these two objectives present significant challenges to the current AI-based designing tools. In this work, we present a method to synthesize body-aware 3D objects from a base mesh given an input body geometry and either text or image as guidance. The generated objects can be simulated on virtual characters, or fabricated for real-world use. We propose to use a mesh deformation procedure that optimizes for both semantic alignment as well as contact and penetration losses. Using our method, users can generate both virtual or real-world objects from text, image, or sketch, without the need for manual artist intervention. We present both qualitative and quantitative results on various object categories, demonstrating the effectiveness of our approach.

Problem

• Synthesizing body-aware 3D objects from input geometry and guidance
• Optimizing mesh deformation for semantic alignment and physical constraints
• Generating functional virtual or real-world objects without manual intervention

Innovation

• Body-aware 3D object synthesis from base mesh
• Mesh deformation optimizing semantic and contact losses
• Generate objects from text, image, or sketch