URDF-Anything+: Autoregressive Articulated 3D Models Generation for Physical Simulation

2026-03-14

AI Summary

Jointly reconstructing the part geometry and kinematic structure of articulated objects from visual inputs remains a significant challenge. This work proposes the first end-to-end autoregressive framework that directly generates complete, executable URDF models from images and object-level 3D cues, sequentially predicting part geometries and joint parameters while automatically inferring the underlying kinematic structure. Notably, this approach achieves single-stage generation of full URDF specifications—a capability not demonstrated by prior methods—and enables the Real-Follow-Sim paradigm, allowing policies trained purely in simulation to be transferred to real robots without any fine-tuning. Evaluated on large-scale benchmarks and real-world tasks, the method substantially outperforms existing approaches in terms of geometric reconstruction fidelity, joint parameter accuracy, and physical executability.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Low Level & Physics-based Vision
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Articulated objects are fundamental for robotics, simulation of physics, and interactive virtual environments. However, reconstructing them from visual input remains challenging, as it requires jointly inferring both part geometry and kinematic structure. We present, an end-to-end autoregressive framework that directly generates executable articulated object models from visual observations. Given image and object-level 3D cues, our method sequentially produces part geometries and their associated joint parameters, resulting in complete URDF models without reliance on multi-stage pipelines. The generation proceeds until the model determines that all parts have been produced, automatically inferring complete geometry and kinematics. Building on this capability, we enable a new Real-Follow-Sim paradigm, where high-fidelity digital twins constructed from visual observations allow policies trained and tested purely in simulation to transfer to real robots without online adaptation. Experiments on large-scale articulated object benchmarks and real-world robotic tasks demonstrate that outperforms prior methods in geometric reconstruction quality, joint parameter accuracy, and physical executability.

Problem

• articulated objects
• 3D reconstruction
• kinematic structure
• visual input
• geometric inference

Innovation

• autoregressive generation
• articulated 3D models
• URDF
• physical simulation
• sim-to-real transfer