SGFormer++: Semantic Graph Transformer for Incremental 3D Scene Graph Generation

2026-06-13

AI Summary

This work addresses the challenges of modeling complex global structures in 3D scene graph generation and catastrophic forgetting in relation category incremental learning by proposing SGFormer++. Built upon a Transformer architecture, SGFormer++ introduces an edge-aware global context fusion mechanism with linear complexity and enhances structural and semantic representations through dedicated graph embedding and semantic injection layers. To mitigate forgetting during incremental learning, it incorporates a spatially guided feature adapter and a cascaded binary prediction head, leveraging parameter-free large language model priors to establish a geometry-aware incremental learning strategy. Evaluated on the 3DSSG benchmark, SGFormer++ achieves state-of-the-art performance, yielding an absolute improvement of 4.49% in Predicate A@1 under incremental settings.

Abstract

In this paper, we propose SGFormer++, a novel Semantic Graph Transformer for 3D scene graph generation (SGG), which aims to parse point cloud scenes into semantic structural graphs, where nodes denote detected object instances and edges encode their pairwise relationships, with the core challenge lying in modeling complex global scene structure. While existing graph convolutional network (GCN)-based methods suffer from over-smoothing and limited receptive fields, SGFormer++ leverages Transformer layers as its backbone to enable global message passing. Specifically, we introduce two key components tailored for 3D SGG: (1) a Graph Embedding Layer++ that efficiently integrates edge-aware global context with linear computational complexity, and (2) a Semantic Injection Layer++ that enriches visual features with linguistic priors from large language models (LLMs) and vision-language models (VLMs), boosting semantic representation without introducing extra trainable parameters. To further address the practical challenge of incremental SGG (I-SGG), where new relationship categories arrive sequentially, we equip SGFormer++ with a novel Spatial-guided Feature Adapter, which calibrates predicate features using subject-object spatial geometry to counter scale variation, and a Cascaded Binary Prediction Head that mitigates catastrophic forgetting via task-incremental classifier expansion and logit distillation. Extensive experiments on the 3DSSG benchmark demonstrate that SGFormer++ achieves state-of-the-art performance in both standard and incremental settings: it yields a significant 4.49% absolute improvement in Predicate A@1 under the incremental setting. Code and data are available at: https://github.com/Andy20178/SGFormer.

Problem

• 3D scene graph generation
• incremental learning
• semantic relationships
• point cloud parsing
• catastrophic forgetting

Innovation

• Semantic Graph Transformer
• Incremental 3D Scene Graph Generation
• Edge-aware Global Context
• Language Model Priors
• Catastrophic Forgetting Mitigation