ES3D: Embedding Semantics into 3D Space for Component-Aware Editing

2026-08-16

AI Summary

This study addresses the limitations of fine-grained semantic understanding and precise component manipulation in existing 3D editing approaches by proposing a novel 3D semantic embedding framework. Leveraging voxelized semantic projection and feature similarity retrieval, the method achieves component-level perception without manual masking. Furthermore, it integrates multi-reference image guidance, text-conditioned control, and generative inpainting mechanisms to enable refined editing. This approach effectively overcomes bottlenecks in specific component retrieval and modification while facilitating robust multimodal conditional control. Consequently, the proposed framework significantly enhances both geometric consistency and semantic coherence in editing outcomes, establishing a new paradigm for the fine-grained generation of 3D assets.

Abstract

Existing 3D editing methods have made notable progress in controllability, yet they remain limited in several important ways. Most approaches rely on text-driven editing, which struggles to express fine-grained visual changes intended by the user. Moreover, many methods require manually supplied 3D masks or introduce unintended changes to regions that should remain untouched. These limitations largely arise from the absence of fine-grained semantic understanding, making it difficult for existing models to retrieve or modify specific 3D components. We introduce ES3D, a framework that embeds semantics directly into 3D space, enabling component-aware retrieval and editing of a 3D asset conditioned on multiple local reference images and optional text queries. We first construct a 3D semantic embedding by projecting multi-view semantic features into the voxelized space of the asset. We then perform 3D component retrieval by computing feature similarity between the 3D semantic embedding and the semantic embeddings of image or text queries. For editing, we employ a pretrained 3D generative model with an inpainting mechanism to modify the retrieved components guided by user-provided images while preserving the rest of the asset. Overall, ES3D is a 3D editing framework that retrieves editable regions based on semantic cues and uses multiple images as conditions. Extensive experiments demonstrate that ES3D produces geometrically consistent and semantically coherent edits, enabling robust image-based and text-assisted control for 3D editing.

Problem

• 3D editing
• fine-grained semantic understanding
• component-aware editing
• text-driven limitations

Innovation

• 3D Semantic Embedding
• Component-Aware Editing
• Voxelized Space
• Multi-view Semantic Features
• Image-conditioned Inpainting