SurGrID: Controllable Surgical Simulation via Scene Graph to Image Diffusion

2025-02-11

AI Summary

Current surgical simulation tools suffer from insufficient visual realism and rigid, non-interactive behavior; moreover, mainstream diffusion models struggle to simultaneously achieve precise controllability and real-time interactivity. To address these limitations, this paper introduces the first scene-graph-to-image diffusion generation framework tailored for surgical training. Our method employs a novel scene-graph pretraining mechanism that explicitly models local–global semantic relationships, integrating graph neural networks with conditional denoising diffusion probabilistic models (DDPMs) to enable structured semantic input-driven, high-fidelity, fine-grained controllable image synthesis. Quantitative evaluation demonstrates significant improvements over state-of-the-art methods in both image quality and scene-graph alignment. Clinical expert assessment further confirms the framework’s high realism and real-time interactive capability, validating its effectiveness in supporting immersive, scenario-based surgical simulation training.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Surgical simulation offers a promising addition to conventional surgical training. However, available simulation tools lack photorealism and rely on hardcoded behaviour. Denoising Diffusion Models are a promising alternative for high-fidelity image synthesis, but existing state-of-the-art conditioning methods fall short in providing precise control or interactivity over the generated scenes. We introduce SurGrID, a Scene Graph to Image Diffusion Model, allowing for controllable surgical scene synthesis by leveraging Scene Graphs. These graphs encode a surgical scene's components' spatial and semantic information, which are then translated into an intermediate representation using our novel pre-training step that explicitly captures local and global information. Our proposed method improves the fidelity of generated images and their coherence with the graph input over the state-of-the-art. Further, we demonstrate the simulation's realism and controllability in a user assessment study involving clinical experts. Scene Graphs can be effectively used for precise and interactive conditioning of Denoising Diffusion Models for simulating surgical scenes, enabling high fidelity and interactive control over the generated content.

Problem

• Enhance surgical simulation realism and control
• Improve image fidelity with Scene Graphs
• Enable interactive surgical scene synthesis

Innovation

• Scene Graph to Image Diffusion
• Pre-training captures local-global info
• Enhances surgical scene fidelity