Holo-Captioning: Toward the Text Equivalent of 3D Scenes

2026-07-02

AI Summary

This work proposes the "holo-captioning" task, which aims to generate structured textual descriptions of 3D scenes that comprehensively capture semantic labels, spatial locations, attributes, and inter-object relationships of all entities. To tackle this challenge, the authors introduce HoloScribe, a novel model featuring an instance-aware decoupled generation pipeline and an anchor-aware instance association mechanism to accurately model relational structures among entities. The study also presents HoloScore, the first dedicated evaluation metric for this task, alongside a large-scale benchmark dataset comprising over 15K scenes and a human-verified test set. Experimental results demonstrate that HoloScribe significantly outperforms existing approaches for dense 3D scene description and state-of-the-art 3D large language models on the proposed benchmark.

Abstract

This work introduces holo-captioning, a novel task that strives to seek the text equivalent of 3D scenes. As the initial step, we formulate holo-captioning as generating a structured textual description that comprehensively depicts all entities within a 3D scene -- including their semantic tags, spatial locations, attributes, and inter-entity relations. To tackle this challenging task, we first develop an effective captioning engine to produce detailed descriptions of individual entity instances and instance pairs, and contribute a large-scale benchmark comprising over 15K scenes for training and evaluation. Building upon this foundation, we propose HoloScribe, a novel model that features an instance-aware decoupled pipeline for generating structured holo-captions, and further incorporates anchor-aware instance linking to identify relational instance pairs. Additionally, we propose a comprehensive evaluation metric named HoloScore, and provide a human-curated test set to ensure reliable model assessment. Experimental results demonstrate that HoloScribe significantly outperforms state-of-the-art 3D dense captioners and 3D LLM generalists, underscoring the effectiveness of our approach. Project page: https://visual-ai.github.io/holocap/

Problem

• holo-captioning
• 3D scene understanding
• structured textual description
• 3D captioning
• scene-text equivalence

Innovation

• holo-captioning
• structured 3D scene description
• instance-aware decoupled pipeline
• anchor-aware instance linking
• HoloScore