UniMesh: Unifying 3D Mesh Understanding and Generation

2026-04-19

AI Summary

Existing 3D vision models typically treat understanding and generation tasks in isolation, leading to inconsistent representations and hindering knowledge transfer. This work proposes UniMesh, a unified framework that jointly learns 3D mesh understanding and generation within a single architecture. By introducing a Mesh Head as a cross-task interface, a Chain of Mesh for semantic-driven iterative editing in a closed loop, and an Actor-Evaluator self-reflection mechanism, UniMesh enables bidirectional enhancement and improves reliability on high-level tasks. Experiments demonstrate that the method achieves state-of-the-art performance on standard benchmarks and exhibits novel capabilities, including mutual reinforcement between understanding and generation and support for user-guided iterative editing.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Recent advances in 3D vision have led to specialized models for either 3D understanding (e.g., shape classification, segmentation, reconstruction) or 3D generation (e.g., synthesis, completion, and editing). However, these tasks are often tackled in isolation, resulting in fragmented architectures and representations that hinder knowledge transfer and holistic scene modeling. To address these challenges, we propose UniMesh, a unified framework that jointly learns 3D generation and understanding within a single architecture. First, we introduce a novel Mesh Head that acts as a cross model interface, bridging diffusion based image generation with implicit shape decoders. Second, we develop Chain of Mesh (CoM), a geometric instantiation of iterative reasoning that enables user driven semantic mesh editing through a closed loop latent, prompting, and re generation cycle. Third, we incorporate a self reflection mechanism based on an Actor Evaluator Self reflection triad to diagnose and correct failures in high level tasks like 3D captioning. Experimental results demonstrate that UniMesh not only achieves competitive performance on standard benchmarks but also unlocks novel capabilities in iterative editing and mutual enhancement between generation and understanding. Code: https://github.com/AIGeeksGroup/UniMesh. Website: https://aigeeksgroup.github.io/UniMesh.

Problem

• 3D understanding
• 3D generation
• unified framework
• knowledge transfer
• holistic scene modeling

Innovation

• UniMesh
• Chain of Mesh
• Mesh Head
• self-reflection mechanism
• unified 3D understanding and generation