SPADE: Spatial-Aware Denoising Network for Open-vocabulary Panoptic Scene Graph Generation with Long- and Local-range Context Reasoning

2025-07-08

AI Summary

Existing open-vocabulary panoramic scene graph generation (PSG) methods rely on pre-trained vision-language models (VLMs), yet suffer from weak spatial relational reasoning, limiting accurate modeling of object-relative positions. To address this, we propose SPADE—the first framework to integrate diffusion model inversion into PSG, leveraging its inherent preservation of image spatial structure to construct a spatially aware denoising network. SPADE further combines lightweight LoRA fine-tuning with a spatially aware relational graph Transformer to jointly model local details and long-range spatial context. On both PSG and Visual Genome benchmarks, SPADE achieves significant improvements in spatial relation prediction accuracy, while remaining compatible with both closed-set and open-vocabulary settings. Our approach establishes a novel paradigm for enhancing VLMs’ spatial modeling capabilities in structured scene understanding.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Panoptic Scene Graph Generation (PSG) integrates instance segmentation with relation understanding to capture pixel-level structural relationships in complex scenes. Although recent approaches leveraging pre-trained vision-language models (VLMs) have significantly improved performance in the open-vocabulary setting, they commonly ignore the inherent limitations of VLMs in spatial relation reasoning, such as difficulty in distinguishing object relative positions, which results in suboptimal relation prediction. Motivated by the denoising diffusion model's inversion process in preserving the spatial structure of input images, we propose SPADE (SPatial-Aware Denoising-nEtwork) framework -- a novel approach for open-vocabulary PSG. SPADE consists of two key steps: (1) inversion-guided calibration for the UNet adaptation, and (2) spatial-aware context reasoning. In the first step, we calibrate a general pre-trained teacher diffusion model into a PSG-specific denoising network with cross-attention maps derived during inversion through a lightweight LoRA-based fine-tuning strategy. In the second step, we develop a spatial-aware relation graph transformer that captures both local and long-range contextual information, facilitating the generation of high-quality relation queries. Extensive experiments on benchmark PSG and Visual Genome datasets demonstrate that SPADE outperforms state-of-the-art methods in both closed- and open-set scenarios, particularly for spatial relationship prediction.

Problem

• Improves spatial relation reasoning in scene graphs
• Enhances open-vocabulary panoptic scene graph generation
• Addresses limitations of vision-language models in spatial understanding

Innovation

• Inversion-guided UNet calibration via LoRA fine-tuning
• Spatial-aware relation graph transformer for context
• Long- and local-range context reasoning integration