Neuro-Symbolic Scene Graph Conditioning for Synthetic Image Dataset Generation

2025-03-21

AI Summary

Synthetic image data often suffers from semantic implausibility and relational inconsistency, limiting the performance of visual reasoning models. Method: This paper introduces the first neuro-symbolic, conditional synthetic data augmentation method tailored for scene graph generation. It explicitly injects structured scene graphs as symbolic constraints into the diffusion model’s generative process, integrating graph neural networks with differentiable symbolic reasoning modules to construct an end-to-end trainable neuro-symbolic image generation framework. Contribution/Results: On scene graph generation, the method significantly narrows the performance gap between synthetic and real data: Recall improves by 2.59% overall and by 2.83% in the unconstrained graph setting. These results validate the efficacy of symbolic guidance in enhancing semantic plausibility and relational consistency of synthetic images, establishing a new paradigm for generating high-quality, interpretable synthetic data for complex visual reasoning tasks.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Machine Learning: Neuro-Symbolic Learning
• Cognitive Modeling & Cognitive Systems: Symbolic Representations

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

As machine learning models increase in scale and complexity, obtaining sufficient training data has become a critical bottleneck due to acquisition costs, privacy constraints, and data scarcity in specialised domains. While synthetic data generation has emerged as a promising alternative, a notable performance gap remains compared to models trained on real data, particularly as task complexity grows. Concurrently, Neuro-Symbolic methods, which combine neural networks' learning strengths with symbolic reasoning's structured representations, have demonstrated significant potential across various cognitive tasks. This paper explores the utility of Neuro-Symbolic conditioning for synthetic image dataset generation, focusing specifically on improving the performance of Scene Graph Generation models. The research investigates whether structured symbolic representations in the form of scene graphs can enhance synthetic data quality through explicit encoding of relational constraints. The results demonstrate that Neuro-Symbolic conditioning yields significant improvements of up to +2.59% in standard Recall metrics and +2.83% in No Graph Constraint Recall metrics when used for dataset augmentation. These findings establish that merging Neuro-Symbolic and generative approaches produces synthetic data with complementary structural information that enhances model performance when combined with real data, providing a novel approach to overcome data scarcity limitations even for complex visual reasoning tasks.

Problem

• Addressing synthetic data performance gap vs real data
• Enhancing synthetic image quality via neuro-symbolic scene graphs
• Improving scene graph generation models through structured constraints

Innovation

• Neuro-Symbolic conditioning enhances synthetic data
• Scene graphs encode relational constraints explicitly
• Combining symbolic and generative methods improves performance