Surgical Gaussian Surfels: Highly Accurate Real-time Surgical Scene Rendering

2025-03-06

AI Summary

To address the challenging problem of geometric reconstruction of deformable soft tissues from monocular endoscopic video—particularly under severe, artifact-free instrument occlusions while preserving fine anatomical details—we propose a surgery-oriented Gaussian Surfel representation. Our method constrains covariance alignment along the viewing direction and defines surface normals via density gradients, eliminating reliance on monocular normal priors. We introduce a view-space position-gradient-driven adaptive splitting mechanism, integrated with surface-aware constraints, anisotropic modeling, a lightweight MLP-based motion field, and local regularization. Evaluated on two in-vivo surgical datasets, our approach achieves state-of-the-art performance in surface accuracy, normal quality, and rendering efficiency, operating at real-time speed (≈30 FPS). The implementation is publicly available.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Learning with Manifolds
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Accurate geometric reconstruction of deformable tissues in monocular endoscopic video remains a fundamental challenge in robot-assisted minimally invasive surgery. Although recent volumetric and point primitive methods based on neural radiance fields (NeRF) and 3D Gaussian primitives have efficiently rendered surgical scenes, they still struggle with handling artifact-free tool occlusions and preserving fine anatomical details. These limitations stem from unrestricted Gaussian scaling and insufficient surface alignment constraints during reconstruction. To address these issues, we introduce Surgical Gaussian Surfels (SGS), which transforms anisotropic point primitives into surface-aligned elliptical splats by constraining the scale component of the Gaussian covariance matrix along the view-aligned axis. We predict accurate surfel motion fields using a lightweight Multi-Layer Perceptron (MLP) coupled with locality constraints to handle complex tissue deformations. We use homodirectional view-space positional gradients to capture fine image details by splitting Gaussian Surfels in over-reconstructed regions. In addition, we define surface normals as the direction of the steepest density change within each Gaussian surfel primitive, enabling accurate normal estimation without requiring monocular normal priors. We evaluate our method on two in-vivo surgical datasets, where it outperforms current state-of-the-art methods in surface geometry, normal map quality, and rendering efficiency, while remaining competitive in real-time rendering performance. We make our code available at https://github.com/aloma85/SurgicalGaussianSurfels

Problem

• Accurate geometric reconstruction of deformable tissues in monocular endoscopic video.
• Handling artifact-free tool occlusions and preserving fine anatomical details.
• Improving surface geometry, normal map quality, and rendering efficiency in real-time.

Innovation

• Surface-aligned elliptical splats for accurate rendering
• Lightweight MLP with locality for tissue deformation
• Homodirectional gradients for fine image detail capture