GARF: Learning Generalizable 3D Reassembly for Real-World Fractures

2025-04-07

AI Summary

This work addresses the poor generalization of models trained on synthetic data when reconstructing real-world fractured objects (e.g., ceramics, bones, eggshells, lithics) from 3D fragments. We propose a fracture-aware pretraining framework coupled with a flow-matching–driven 6-DoF alignment mechanism and a one-step pre-assembly strategy, significantly enhancing robustness to unseen shapes and variable fragment counts. We introduce Fractura—the first benchmark dataset capturing realistic fracture patterns. Our method outperforms state-of-the-art approaches in both synthetic and real-world settings: rotation error is reduced by 82.87%, and part-matching accuracy improves by 25.15%. The core innovation lies in embedding physically grounded fracture priors into representation learning and achieving end-to-end, geometrically consistent fragment registration and assembly.

Technology Category

• Computer Vision: Segmentation
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

3D reassembly is a challenging spatial intelligence task with broad applications across scientific domains. While large-scale synthetic datasets have fueled promising learning-based approaches, their generalizability to different domains is limited. Critically, it remains uncertain whether models trained on synthetic datasets can generalize to real-world fractures where breakage patterns are more complex. To bridge this gap, we propose GARF, a generalizable 3D reassembly framework for real-world fractures. GARF leverages fracture-aware pretraining to learn fracture features from individual fragments, with flow matching enabling precise 6-DoF alignments. At inference time, we introduce one-step preassembly, improving robustness to unseen objects and varying numbers of fractures. In collaboration with archaeologists, paleoanthropologists, and ornithologists, we curate Fractura, a diverse dataset for vision and learning communities, featuring real-world fracture types across ceramics, bones, eggshells, and lithics. Comprehensive experiments have shown our approach consistently outperforms state-of-the-art methods on both synthetic and real-world datasets, achieving 82.87% lower rotation error and 25.15% higher part accuracy. This sheds light on training on synthetic data to advance real-world 3D puzzle solving, demonstrating its strong generalization across unseen object shapes and diverse fracture types.

Problem

• Generalizing 3D reassembly from synthetic to real-world fractures
• Handling complex breakage patterns in real-world objects
• Improving robustness for unseen objects and fracture variations

Innovation

• Fracture-aware pretraining for fragment feature learning
• Flow matching for precise 6-DoF alignments
• One-step preassembly for unseen object robustness