ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation

2026-07-07

AI Summary

Existing unified 3D foundation models suffer from entangled structural cues and fine-grained details during text–3D geometric interaction due to the absence of an explicit, multi-scale semantic alignment mechanism, thereby limiting both generation and understanding performance. To address this, we propose an elastic semantic anchoring mechanism that constructs multi-granularity geometric representations via a scale-aware octree tokenizer and introduces anchor tokens paired with a lightweight per-block router to enable sparse, on-demand cross-modal alignment at matched levels of abstraction. Integrated into a unified Transformer architecture, our approach supports end-to-end multi-task training and achieves state-of-the-art results on both image/text-to-3D generation and 3D captioning tasks, while reducing FLOPs and inference latency by approximately 50% compared to its non-elastic counterpart.

Abstract

Unified 3D foundation models aspire to generate 3D assets and reason about them in language within a single backbone, but their text-3D interaction remains largely implicit. Existing methods concatenate text and 3D tokens into a flat sequence and rely on self-attention, collapsing coarse structural cues and fine geometric details into one undifferentiated representation. We introduce ELSA3D, a unified 3D model that addresses this with elastic semantic anchoring, structuring language and geometric reasoning jointly along matched abstraction scales. ELSA3D represents geometry with a scale-aware octree tokenizer and introduces Anchor Tokens, sparse cross-modal units that select semantic cues, route them to the most relevant 3D scale, retrieve scale-specific geometric evidence, and write the fused signal back into the unified representation, keeping interaction sparse yet precise. A lightweight per-block router makes both computation and reasoning elastic, choosing which text tokens instantiate anchors at which geometric scale so that cross-modal capacity concentrates where alignment is most needed. ELSA3D achieves state-of-the-art performance across image-to-3D generation, text-to-3D generation, and 3D captioning, outperforming the strongest unified baseline while roughly halving FLOPs and inference latency relative to the non-elastic version of the same model.

Problem

• unified 3D foundation models
• text-3D interaction
• cross-modal alignment
• geometric representation
• semantic anchoring

Innovation

• elastic semantic anchoring
• anchor tokens
• scale-aware octree
• unified 3D foundation model
• sparse cross-modal interaction