VecSet-Edit: Unleashing Pre-trained LRM for Mesh Editing from Single Image

2026-02-04

AI Summary

This work addresses the limitations of existing single-image-driven 3D mesh editing methods, which suffer from low-fidelity local edits due to coarse voxel representations and reliance on labor-intensive 3D mask annotations. To overcome these challenges, we introduce, for the first time, the high-fidelity VecSet Large Reconstruction Model (LRM) into the mesh editing task. By leveraging the spatial characteristics of its internal tokens, we propose a 2D-guided editing framework that eliminates the need for 3D masks. Our approach features three key innovations: Mask-guided Token Seeding, Attention-aligned Token Gating, and Drift-aware Token Pruning, complemented by a detail-preserving texture baking technique. Using only a single input image and a 2D mask, our method enables high-fidelity, locally controllable 3D mesh editing while preserving the original geometry and texture details.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

3D editing has emerged as a critical research area to provide users with flexible control over 3D assets. While current editing approaches predominantly focus on 3D Gaussian Splatting or multi-view images, the direct editing of 3D meshes remains underexplored. Prior attempts, such as VoxHammer, rely on voxel-based representations that suffer from limited resolution and necessitate labor-intensive 3D mask. To address these limitations, we propose \textbf{VecSet-Edit}, the first pipeline that leverages the high-fidelity VecSet Large Reconstruction Model (LRM) as a backbone for mesh editing. Our approach is grounded on a analysis of the spatial properties in VecSet tokens, revealing that token subsets govern distinct geometric regions. Based on this insight, we introduce Mask-guided Token Seeding and Attention-aligned Token Gating strategies to precisely localize target regions using only 2D image conditions. Also, considering the difference between VecSet diffusion process versus voxel we design a Drift-aware Token Pruning to reject geometric outliers during the denoising process. Finally, our Detail-preserving Texture Baking module ensures that we not only preserve the geometric details of original mesh but also the textural information. More details can be found in our project page: https://github.com/BlueDyee/VecSet-Edit/tree/main

Problem

• 3D mesh editing
• single-image 3D reconstruction
• Large Reconstruction Model
• geometric detail preservation
• 3D asset manipulation

Innovation

• VecSet
• mesh editing
• Large Reconstruction Model
• token-level control
• single-image 3D editing