SPEAR: A Simulator for Photorealistic Embodied AI Research

2026-07-07

AI Summary

Existing high-fidelity simulators struggle to meet the demands of embodied intelligence research in terms of generality, programmability, and rendering efficiency. To address this gap, this work proposes SPEAR—a Python-based simulation library built on Unreal Engine—that exposes over 14,000 UE functions to Python for the first time, enabling unprecedented procedural control. SPEAR employs a modular plugin architecture and a high-level task graph scheduling model to support deterministic execution of complex tasks, delivering high-definition multimodal images along with accurate annotations to NumPy arrays at an efficient rate of 73 FPS. The system has been successfully applied to diverse tasks including agent control, city-scale rendering, multi-view facial imaging, physics-informed co-simulation, and language-driven scene editing, significantly enhancing both simulation efficiency and functional coverage.

Abstract

Interactive simulators have become powerful tools for training embodied agents and generating synthetic visual data, but existing photorealistic simulators suffer from limited generality, programmability, and rendering speed. We address these limitations by introducing SPEAR: A Simulator for Photorealistic Embodied AI Research. At its core, SPEAR is a Python library that can connect to, and programmatically control, any Unreal Engine (UE) application via a modular plugin architecture. SPEAR exposes over 14K unique UE functions to Python, representing an order-of-magnitude increase in programmable functionality over existing UE-based simulators. Additionally, a single SPEAR instance can render 1920x1080 photorealistic beauty images directly into a user's NumPy array at 73 frames per second - an order of magnitude faster than existing UE plugins - while also providing ground truth image modalities that are not available in any existing UE-based simulator (e.g., a non-diffuse intrinsic image decomposition, material IDs, and physically based shading parameters). Finally, SPEAR introduces an expressive high-level programming model that enables users to specify complex graphs of UE work with arbitrary data dependencies among work items, and to execute these graphs deterministically within a single UE frame. We demonstrate the utility of SPEAR through a diverse collection of example applications: controlling multiple embodied agents with distinct action spaces (e.g., humans, cars, and robots) across several in-the-wild UE projects; rendering photorealistic city-scale environments; manipulating UE's procedural content generation systems; rendering synchronized multi-view images of detailed human faces; coordinating an interactive co-simulation with the MuJoCo physics simulator; and editing scenes with natural language via an AI coding assistant.

Problem

• photorealistic simulation
• embodied AI
• Unreal Engine
• programmability
• rendering speed

Innovation

• photorealistic simulation
• Unreal Engine integration
• programmable embodied AI
• high-speed rendering
• intrinsic image decomposition