SPAFormer: Sequential 3D Part Assembly with Transformers

2024-03-09

arXiv.org

AI Summary

To address the combinatorial explosion problem in 3D part assembly (3D-PA) caused by increasing part counts, this paper proposes a grammar-guided sequential modeling paradigm that integrates parallel generation with autoregressive decoding for stepwise, precise prediction of part poses. We design a part-attribute and ordering-knowledge enhancement mechanism to explicitly model implicit assembly rules. Furthermore, we introduce PartNet-Assembly—the first high-difficulty benchmark for 3D-PA. Our Transformer-based method incorporates weakly supervised sequence modeling, knowledge-enhanced embeddings, and multi-task joint training. It achieves significant improvements over prior work in long-range assembly reasoning and cross-task generalization. Code is publicly available.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We introduce SPAFormer, an innovative model designed to overcome the combinatorial explosion challenge in the 3D Part Assembly (3D-PA) task. This task requires accurate prediction of each part's poses in sequential steps. As the number of parts increases, the possible assembly combinations increase exponentially, leading to a combinatorial explosion that severely hinders the efficacy of 3D-PA. SPAFormer addresses this problem by leveraging weak constraints from assembly sequences, effectively reducing the solution space's complexity. Since the sequence of parts conveys construction rules similar to sentences structured through words, our model explores both parallel and autoregressive generation. We further strengthen SPAFormer through knowledge enhancement strategies that utilize the attributes of parts and their sequence information, enabling it to capture the inherent assembly pattern and relationships among sequentially ordered parts. We also construct a more challenging benchmark named PartNet-Assembly covering 21 varied categories to more comprehensively validate the effectiveness of SPAFormer. Extensive experiments demonstrate the superior generalization capabilities of SPAFormer, particularly with multi-tasking and in scenarios requiring long-horizon assembly. Code is available at https://github.com/xuboshen/SPAFormer.

Problem

• Overcomes combinatorial explosion in 3D Part Assembly
• Predicts part poses in sequential assembly steps
• Reduces solution space complexity using sequence constraints

Innovation

• Transformers for 3D assembly
• Weak constraints reduce complexity
• Parallel and autoregressive generation methods