Semantic Radiance Fields as Simulators for Spatial Reasoning in Real-World Scenes

2026-08-13

AI Summary

This work addresses the lack of simulation environments that simultaneously offer geometric realism and semantic ground truth for training embodied agents in spatial reasoning. The authors propose a unified simulator based on Semantic Radiance Fields (SRF), which elevates 2D semantic segmentation from pretrained vision models into 3D radiance fields that jointly encode geometry, appearance, and category-level semantics. This framework enables high-fidelity reconstruction of real-world scenes with queryable semantic information, supporting novel view synthesis, free-space navigation, and semantic queries, while integrating a physics engine to facilitate agent-environment interaction. Evaluated on an apple-harvesting task in orchards, the system demonstrates superior rendering quality, accurate semantic labeling, and robust physical interaction, establishing an effective and realistic platform for training and evaluating embodied agents in spatial reasoning tasks.

Abstract

Training and evaluating spatial reasoning in embodied agents requires diverse environments that are both geometrically faithful and semantically queryable. Synthetic simulators offer ground truth semantics but sacrifice realism; simulators based on reconstructions of real-world environments have realistic appearance but lack ground truth semantics by default. We propose using Semantic Radiance Fields (SRF) as simulators for spatial reasoning agents. SRFs are a representation that unifies these requirements by lifting 2D semantic segmentations from pretrained vision models into a 3D radiance field that jointly encodes geometry, appearance, and per-class semantic identity. The resulting fields are reconstructed from posed RGB captures of real scenes and support novel-view synthesis, semantic and free-space queries within a single grounded representation. This enables the efficient generation of diverse real-world environments to train and evaluate spatial reasoning models. As an example application, we outline an SRF-driven simulator for an orchard apple-reaching task, in which the radiance field supplies camera rendering, semantic ground truth, and occupancy queries to a physics engine.

Problem

• spatial reasoning
• semantic simulation
• real-world scenes
• embodied agents
• ground truth semantics

Innovation

• Semantic Radiance Fields
• spatial reasoning
• 3D scene representation
• semantic segmentation
• embodied simulation