PuzzleFusion++: Auto-agglomerative 3D Fracture Assembly by Denoise and Verify

2024-06-01

arXiv.org

AI Summary

This work addresses the fully automatic 3D reassembly of fractured artifacts/objects by proposing an iterative auto-agglomerative assembly paradigm inspired by human spatial reasoning. Methodologically, it introduces the first application of diffusion models for 6-DoF pose denoising estimation, integrated with a Transformer-driven geometric-semantic alignment verifier to enable end-to-end, iterative hierarchical clustering. Pose estimation and merging decisions are jointly optimized via Chamfer distance. On the Breaking Bad benchmark, the method achieves over 10% higher part-matching accuracy and 50% lower Chamfer distance than prior state-of-the-art, significantly improving reconstruction robustness and fidelity on complex fracture surfaces. The core contribution is the first diffusion-Transformer co-designed auto-agglomerative framework, enabling closed-loop optimization from fragment perception to globally consistent 3D modeling.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: State Estimation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Agentic search

Abstract

This paper proposes a novel"auto-agglomerative"3D fracture assembly method, PuzzleFusion++, resembling how humans solve challenging spatial puzzles. Starting from individual fragments, the approach 1) aligns and merges fragments into larger groups akin to agglomerative clustering and 2) repeats the process iteratively in completing the assembly akin to auto-regressive methods. Concretely, a diffusion model denoises the 6-DoF alignment parameters of the fragments simultaneously, and a transformer model verifies and merges pairwise alignments into larger ones, whose process repeats iteratively. Extensive experiments on the Breaking Bad dataset show that PuzzleFusion++ outperforms all other state-of-the-art techniques by significant margins across all metrics, in particular by over 10% in part accuracy and 50% in Chamfer distance. The code will be available on our project page: https://puzzlefusion-plusplus.github.io.

Problem

• Auto-agglomerative 3D fracture assembly
• Denoise 6-DoF alignment parameters
• Verify and merge pairwise alignments

Innovation

• Auto-agglomerative 3D fracture assembly
• Denoising with diffusion model
• Verifying with transformer model