SG-Tailor: Inter-Object Commonsense Relationship Reasoning for Scene Graph Manipulation

2025-03-23

AI Summary

This paper addresses structural conflicts and commonsense violations arising from node addition and edge modification in dynamic scene graph editing. To this end, we propose a conflict-aware autoregressive relation prediction framework. Methodologically: (1) we introduce a Cut-And-Stitch mechanism that ensures global graph structural coherence following local edge edits; (2) we pioneer cross-graph commonsense edge generation for newly added nodes by explicitly injecting external commonsense knowledge to enhance semantic plausibility. Compared to prior approaches, our method achieves state-of-the-art performance across multiple scene graph editing benchmarks, significantly improving both edit consistency and commonsense compliance. Moreover, it serves as a plug-and-play module that boosts downstream tasks—including scene generation and robotic manipulation—without architectural modifications.

Technology Category

• Knowledge Representation and Reasoning: Common-Sense Reasoning
• Computer Vision: Visual Reasoning & Symbolic Representations
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Scene graphs capture complex relationships among objects, serving as strong priors for content generation and manipulation. Yet, reasonably manipulating scene graphs -- whether by adding nodes or modifying edges -- remains a challenging and untouched task. Tasks such as adding a node to the graph or reasoning about a node's relationships with all others are computationally intractable, as even a single edge modification can trigger conflicts due to the intricate interdependencies within the graph. To address these challenges, we introduce SG-Tailor, an autoregressive model that predicts the conflict-free relationship between any two nodes. SG-Tailor not only infers inter-object relationships, including generating commonsense edges for newly added nodes but also resolves conflicts arising from edge modifications to produce coherent, manipulated graphs for downstream tasks. For node addition, the model queries the target node and other nodes from the graph to predict the appropriate relationships. For edge modification, SG-Tailor employs a Cut-And-Stitch strategy to solve the conflicts and globally adjust the graph. Extensive experiments demonstrate that SG-Tailor outperforms competing methods by a large margin and can be seamlessly integrated as a plug-in module for scene generation and robotic manipulation tasks.

Problem

• Manipulating scene graphs without causing conflicts
• Predicting conflict-free relationships between object nodes
• Resolving edge modification conflicts for coherent graphs

Innovation

• Autoregressive model predicts conflict-free relationships
• Cut-And-Stitch strategy resolves edge modification conflicts
• Query-based node addition for commonsense edge generation