LoST: Level of Semantics Tokenization for 3D Shapes

2026-03-18

AI Summary

Existing 3D shape tokenization methods rely on rendering- or compression-oriented geometric hierarchies, leading to semantically incoherent representations and inefficient token usage in autoregressive modeling. This work proposes a semantic-driven tokenization framework that prioritizes 3D shape tokens according to semantic saliency, enabling the generation of structurally complete shapes from early tokens while progressively refining geometric and semantic details with subsequent ones. To achieve this, we introduce the Relational Inter-Distance Alignment (RIDA) loss, which aligns the relational structure of the 3D latent space with that of DINO-derived semantic feature space. Our approach significantly outperforms existing methods in both geometric and semantic reconstruction metrics, achieving high-quality generation with only 0.1%–10% of the tokens and effectively supporting downstream tasks such as semantic retrieval.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Tokenization is a fundamental technique in the generative modeling of various modalities. In particular, it plays a critical role in autoregressive (AR) models, which have recently emerged as a compelling option for 3D generation. However, optimal tokenization of 3D shapes remains an open question. State-of-the-art (SOTA) methods primarily rely on geometric level-of-detail (LoD) hierarchies, originally designed for rendering and compression. These spatial hierarchies are often token-inefficient and lack semantic coherence for AR modeling. We propose Level-of-Semantics Tokenization (LoST), which orders tokens by semantic salience, such that early prefixes decode into complete, plausible shapes that possess principal semantics, while subsequent tokens refine instance-specific geometric and semantic details. To train LoST, we introduce Relational Inter-Distance Alignment (RIDA), a novel 3D semantic alignment loss that aligns the relational structure of the 3D shape latent space with that of the semantic DINO feature space. Experiments show that LoST achieves SOTA reconstruction, surpassing previous LoD-based 3D shape tokenizers by large margins on both geometric and semantic reconstruction metrics. Moreover, LoST achieves efficient, high-quality AR 3D generation and enables downstream tasks like semantic retrieval, while using only 0.1%-10% of the tokens needed by prior AR models.

Problem

• 3D shape tokenization
• semantic coherence
• autoregressive modeling
• level-of-detail
• generative modeling

Innovation

• semantic tokenization
• autoregressive 3D generation
• relational alignment
• DINO features
• 3D shape representation