Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot

2026-01-05

arXiv.org

AI Summary

Existing simulation platforms struggle to support effective sim-to-real transfer due to fragmentation, limited scenario diversity, or insufficient fidelity. To address this, this work presents a unified, high-fidelity humanoid robot simulation platform featuring a novel large language model (LLM)-driven natural language interface for procedural scene generation and an automated evaluation pipeline powered by vision-language models (VLMs). The platform enables large-scale synthesis of diverse manipulation scenarios and policy training data, accompanied by an open-sourced dataset comprising over 10,000 hours of synthetic experience. Experimental results demonstrate that, under controlled conditions, the synthetic data can effectively substitute real-world data for scalable policy training and achieve robust zero-shot sim-to-real transfer.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The development of robust and generalizable robot learning models is critically contingent upon the availability of large-scale, diverse training data and reliable evaluation benchmarks. Collecting data in the physical world poses prohibitive costs and scalability challenges, and prevailing simulation benchmarks frequently suffer from fragmentation, narrow scope, or insufficient fidelity to enable effective sim-to-real transfer. To address these challenges, we introduce Genie Sim 3.0, a unified simulation platform for robotic manipulation. We present Genie Sim Generator, a large language model (LLM)-powered tool that constructs high-fidelity scenes from natural language instructions. Its principal strength resides in rapid and multi-dimensional generalization, facilitating the synthesis of diverse environments to support scalable data collection and robust policy evaluation. We introduce the first benchmark that pioneers the application of LLM for automated evaluation. It leverages LLM to mass-generate evaluation scenarios and employs Vision-Language Model (VLM) to establish an automated assessment pipeline. We also release an open-source dataset comprising more than 10,000 hours of synthetic data across over 200 tasks. Through systematic experimentation, we validate the robust zero-shot sim-to-real transfer capability of our open-source dataset, demonstrating that synthetic data can server as an effective substitute for real-world data under controlled conditions for scalable policy training. For code and dataset details, please refer to: https://github.com/AgibotTech/genie_sim.

Problem

• robot learning
• simulation fidelity
• sim-to-real transfer
• training data scalability
• evaluation benchmark

Innovation

• LLM-powered simulation
• high-fidelity synthetic data
• automated evaluation benchmark
• sim-to-real transfer
• Vision-Language Model (VLM)