Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modeling

2025-02-04

AI Summary

Modeling articulated 3D objects under open-vocabulary settings remains challenging—existing approaches are constrained by predefined categories and reliance on manually annotated data, limiting generalization to diverse rigid objects such as tools and toys. Method: We propose the first end-to-end, category-agnostic framework for automatic articulated object construction. Our approach integrates multimodal vision-language models (VLMs) with visual prompt learning to achieve semantic-driven, geometry-aware part segmentation and functional joint parameterization. It enables direct generation of structurally sound, semantically coherent, and kinematically accurate articulated models from a single rigid 3D mesh. Contributions/Results: (1) Eliminates the closed-category assumption, enabling true open-vocabulary modeling; (2) Constructs a large-scale, high-quality articulated object asset library, substantially expanding coverage beyond existing datasets; (3) Policies trained on simulated embodiments of our generated models successfully transfer to real robotic systems.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

3D articulated objects modeling has long been a challenging problem, since it requires to capture both accurate surface geometries and semantically meaningful and spatially precise structures, parts, and joints. Existing methods heavily depend on training data from a limited set of handcrafted articulated object categories (e.g., cabinets and drawers), which restricts their ability to model a wide range of articulated objects in an open-vocabulary context. To address these limitations, we propose Articulate Anymesh, an automated framework that is able to convert any rigid 3D mesh into its articulated counterpart in an open-vocabulary manner. Given a 3D mesh, our framework utilizes advanced Vision-Language Models and visual prompting techniques to extract semantic information, allowing for both the segmentation of object parts and the construction of functional joints. Our experiments show that Articulate Anymesh can generate large-scale, high-quality 3D articulated objects, including tools, toys, mechanical devices, and vehicles, significantly expanding the coverage of existing 3D articulated object datasets. Additionally, we show that these generated assets can facilitate the acquisition of new articulated object manipulation skills in simulation, which can then be transferred to a real robotic system. Our Github website is https://articulate-anymesh.github.io.

Problem

• Modeling 3D articulated objects accurately and semantically.
• Overcoming limitations of existing category-restricted methods.
• Generating high-quality articulated objects for robotics simulation.

Innovation

• Automates rigid 3D mesh articulation
• Uses Vision-Language Models semantics
• Expands 3D articulated datasets coverage