Handle-based Mesh Deformation Guided By Vision Language Model

2025-06-05

AI Summary

Existing mesh deformation methods suffer from low output quality, heavy reliance on manual parameter tuning, or data-intensive training. This paper proposes a training-free, handle-based mesh deformation framework: a vision-language model (VLM) parses natural-language instructions to automatically identify key deformation handles and movable parts; conical singularity detection—introduced for the first time—and multi-view geometric consistency voting jointly mitigate VLM-induced uncertainty in screen-space handle pose estimation. Evaluated on CLIP (+18.2%) and GPTEval3D (+22.7%), our method achieves superior alignment with user intent while reducing membrane energy distortion by 34.5%, thereby balancing semantic understanding accuracy and geometric fidelity.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Mesh deformation is a fundamental tool in 3D content manipulation. Despite extensive prior research, existing approaches often suffer from low output quality, require significant manual tuning, or depend on data-intensive training. To address these limitations, we introduce a training-free, handle-based mesh deformation method. % Our core idea is to leverage a Vision-Language Model (VLM) to interpret and manipulate a handle-based interface through prompt engineering. We begin by applying cone singularity detection to identify a sparse set of potential handles. The VLM is then prompted to select both the deformable sub-parts of the mesh and the handles that best align with user instructions. Subsequently, we query the desired deformed positions of the selected handles in screen space. To reduce uncertainty inherent in VLM predictions, we aggregate the results from multiple camera views using a novel multi-view voting scheme. % Across a suite of benchmarks, our method produces deformations that align more closely with user intent, as measured by CLIP and GPTEval3D scores, while introducing low distortion -- quantified via membrane energy. In summary, our approach is training-free, highly automated, and consistently delivers high-quality mesh deformations.

Problem

• Improves handle-based mesh deformation quality without training
• Reduces manual tuning via vision-language model guidance
• Minimizes distortion in automated 3D content manipulation

Innovation

• Training-free handle-based mesh deformation method
• Vision-Language Model for handle selection
• Multi-view voting reduces prediction uncertainty