AnnotateAnything: Automatic Annotation of 3D Assets for Robot Manipulation

2026-06-15

AI Summary

Existing 3D assets often lack semantic and physical interaction knowledge, hindering robots’ ability to understand where and how to manipulate objects. This work proposes AnnotateAnything, a framework that for the first time enables large-scale, fully automatic, and unified generation of multi-type robotic manipulation annotations. By integrating vision-language reasoning with physical constraints, the method combines geometric optimization, executable trajectory generation, and large-scale parallel physics simulation to efficiently produce structured, diverse, and physically feasible manipulation annotations. Experiments demonstrate that the framework substantially improves annotation efficiency and task success rates, while effectively supporting downstream applications such as functional region detection, robotic visual question answering, and vision-based instruction fine-tuning.

Abstract

Simulation enables scalable robot data collection, but raw 3D assets provide only geometry, lacking the semantic, interactive, and physical knowledge needed to specify where and how robots should act. In this work, we present AnnotateAnything, a general automatic annotation framework that converts passive 3D assets into manipulation-ready assets with structured, diverse, and executable manipulation labels. AnnotateAnything is built around two complementary pipelines. First, a unified visual-language annotation pipeline using vision-language reasoning to infer object semantics, interaction constraints, and 3D-grounded cues, providing human-prior guidance for identifying meaningful interaction regions. Second, a fully automatic and massively parallel physics annotation pipeline grounds these priors in each asset's geometry and physical constraints through candidate generation, geometry optimization and trajectory generation. This pipeline produces diverse and executable action annotations, including grasp poses, dexterous contacts, articulation waypoints, insertion directions, hanging affordances, and navigation targets. Using the generated annotations, we further build an asynchronous parallel simulation data-collection system across diverse objects, tasks, and robot embodiments. Experiments demonstrate that AnnotateAnything achieves superior annotation efficiency, data-collection efficiency, and task success rates over existing annotation and data-generation pipelines, while also supporting downstream tasks such as affordance detection, robotic VQA, and visual instruction finetuning. We provide project materials on the project page and plan to release the full code, annotations, and benchmark to facilitate future research. Videos, code, demo assets, and annotations are provided in supplementary materials Project page: https://tourmaline-caramel-169490.netlify.app.

Problem

• 3D asset annotation
• robot manipulation
• semantic knowledge
• physical affordances
• interactive labeling

Innovation

• automatic annotation
• 3D asset manipulation
• vision-language reasoning
• physics-based simulation
• robotic affordance