R3D: Quantitative 3D Spatial Reasoning for Egocentric Wearables

2026-07-02

AI Summary

Existing benchmarks struggle to effectively evaluate the quantitative 3D spatial reasoning capabilities of wearable devices under natural first-person RGB-D video settings. To address this gap, this work introduces R3D-Bench—the first multi-task quantitative 3D spatial question-answering benchmark tailored for such scenarios—and proposes a model-agnostic R3D reasoning framework. The framework reconstructs 3D object representations via instance segmentation and depth lifting, then supplies structured spatial information to large language models (e.g., Qwen3-VL 235B) through eight composable spatial tools, thereby avoiding direct embedding of raw 3D data. Experimental results demonstrate that R3D achieves an average relative accuracy of 73.5% on this benchmark, significantly outperforming depth-aware (61.9%) and RGB-only (46.5%) baselines.

Abstract

Quantitative 3D spatial reasoning from egocentric RGB-D video is a critical capability for next-generation wearable assistants. Yet existing benchmarks do not reflect the challenges of handling (1) natural egocentric video, (2) posed RGB-D video inputs, and (3) challenging quantitative 3D spatial reasoning Q&A. To fill this gap, we introduce R3D-Bench (Reasoning in 3D), a benchmark of 3,033 quantitative spatial reasoning questions across 15 types -- spanning multiple-choice, distance-based, and volumetric reasoning questions -- built on top of 57 egocentric video sequences from Aria Digital Twin. To set a strong baseline on this dataset, we introduce R3D, a model-agnostic spatial tool-calling framework. In contrast to existing approaches that directly embed 3D information into the model's input representation, R3D constructs a 3D scene from video using segmentation and depth-lifted object representations. It provides this information to an LLM through eight composable spatial tools. On R3D-Bench, R3D with Qwen3-VL 235B achieves 73.5% mean relative accuracy, substantially outperforming the best depth-enabled baseline (CuTR+Tools, 61.9%) and the best RGB-only baseline (Gemini 3 Flash, 46.5%).

Problem

• egocentric video
• RGB-D
• 3D spatial reasoning
• quantitative reasoning
• wearable assistants

Innovation

• 3D spatial reasoning
• egocentric RGB-D video
• model-agnostic framework
• spatial tool-calling
• quantitative QA benchmark