MagicSim: A Unified Infrastructure for Executable Embodied Interaction

2026-06-16

AI Summary

Existing simulation systems struggle to cohesively support the integration of control, skills, and planning, often relying on non-reproducible “magic” actions or fragmented environments. This work proposes a unified embodied interaction framework grounded in a deterministic batched runtime and a shared Markov Decision Process (MDP). By adopting a YAML-first specification, the framework decouples content, behavior, and agent interfaces, and establishes an execution pipeline—Command → Skill → Planner → Robot → Record—that, for the first time, unifies task definition, trajectory collection, agent interaction, evaluation, and planner integration within a closed-loop runtime. The system enables consistent simulation across diverse tasks, physical rules, sensors, and robot morphologies, automatically generating high-quality multimodal trajectories with aligned language, actions, vision, and task states.

Abstract

Robot learning and embodied agents now require simulation to serve as a shared execution substrate linking control, skills, and planning, not only as a renderer, controller testbed, or fixed task environment. Existing pipelines split these layers with "magic" actions, disconnected training environments, or forward-only renders that cannot reproduce, evaluate, and annotate the same episode. We present MagicSim, an embodied interaction infrastructure built around one deterministic batched runtime and a shared Markov decision process (MDP). From YAML-first specifications that decouple contents, placement, behavior, and agent exposure, MagicSim constructs diverse executable worlds spanning task families, interaction regimes, physics, layouts, sensors, avatars, and robot embodiments in one reset-and-step loop. A common execution interface grounds high-level commands through controllers, atomicskills, planner primitives, and asynchronous planning, realizing them as robot actions rather than simulator-side state edits. One task definition supports three capabilities: benchmark and RL evaluation, an autocollect interface that automatically turns commands into grounded trajectories, and agent/VLM-facing interaction. For automatic execution, commands flow through a Command->Skill->Planner->Robot->Record pipeline, while per-environment command, skill, planning, retry, annotation, and episode states advance independently above the shared physics tick. Successful rollouts are saved as structured multimodal trajectories aligning language supervision, action representations, visual/geometric representations, and task-level status with the executed episode. MagicSim thus unifies diverse world construction, embodied execution, task evaluation, automatic rollout generation, and interactive agent interfaces in one planner-in-the-loop runtime.

Problem

• embodied interaction
• simulation infrastructure
• executable simulation
• robot learning
• Markov decision process

Innovation

• embodied interaction
• unified simulation infrastructure
• executable MDP
• automatic trajectory generation
• planner-in-the-loop