ReRef-3D: A Benchmark for Spatial Referring Expression-Guided 3D Scene Rearrangement

2026-08-16

AI Summary

This study addresses the absence of standardized evaluation benchmarks for language-guided 3D object placement by constructing a specialized dataset comprising 33,000 instructions. It introduces a novel region-based evaluation mechanism with multi-hop referencing, integrating dual verification metrics for both relational satisfaction and physical validity. Experiments demonstrate that fine-tuned LLaVA-3D achieves a valid placement rate of 68.3%. By filling this critical evaluation gap, this work establishes a reliable benchmark for 3D spatial reasoning and reveals that current models exhibit significantly stronger capabilities in understanding semantic relationships than in adhering to physical constraints, offering new insights into the field’s developmental trajectory.

Abstract

We introduce ReRef-3D, a benchmark for language-guided placement in 3D scenes. It contains 33,826 instructions across 998 CLEVR-derived scenes, spanning 16 placement families and direct, one-hop, and two-hop references. Each instruction must be resolved into a valid new placement position. Given that an instruction defines a region of acceptable placements rather than one coordinate, our evaluation inserts a prediction into the scene, recomputes relations, and tests relation satisfaction and physical validity. Each instruction also includes a verified naturalized rewrite. After fine-tuning, LLaVA-3D, 3D-LLM, and PlaceIt3D produce valid placements for 68.3%, 31.6%, and 22.4% of instructions, respectively. Across models, relation satisfaction surpasses physical validity, relations such as nearest and between are the most difficult, and phrasing has minimal effect on performance.

Problem

• 3D scene rearrangement
• spatial referring expression
• language-guided placement
• benchmark

Innovation

• 3D Scene Rearrangement
• Spatial Referring Expression
• Benchmark
• Language-Guided Placement
• Dynamic Evaluation