Synthetic Dataset Generation and Validation for Robotic Surgery Instrument Segmentation

2026-02-14

AI Summary

This work addresses the scarcity and high cost of real annotated data for robotic surgical instrument segmentation by proposing a scalable and reproducible synthetic data generation and validation framework. Leveraging a meticulously reconstructed Da Vinci robotic arm model, the framework automatically generates photorealistic, fully annotated video sequences in Autodesk Maya, incorporating randomized motion, dynamic lighting, and synthetic blood textures, alongside physically based rendering and pixel-accurate mask generation. Through systematic evaluation of the impact of synthetic data proportions on segmentation performance, the study reveals—for the first time—that a moderate blend of real and synthetic data significantly enhances model generalization, whereas excessive reliance on synthetic data induces domain shift. Experimental results demonstrate that an appropriately balanced mixed-training strategy outperforms training on real data alone, achieving superior segmentation performance in real-world surgical scenarios.

Technology Category

• Computer Vision: Segmentation
• Intelligent Robots: Manipulation
• Natural Language Processing: Safety and Robustness

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Web data generation and simulation
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

This paper presents a comprehensive workflow for generating and validating a synthetic dataset designed for robotic surgery instrument segmentation. A 3D reconstruction of the Da Vinci robotic arms was refined and animated in Autodesk Maya through a fully automated Python-based pipeline capable of producing photorealistic, labeled video sequences. Each scene integrates randomized motion patterns, lighting variations, and synthetic blood textures to mimic intraoperative variability while preserving pixel-accurate ground truth masks. To validate the realism and effectiveness of the generated data, several segmentation models were trained under controlled ratios of real and synthetic data. Results demonstrate that a balanced composition of real and synthetic samples significantly improves model generalization compared to training on real data only, while excessive reliance on synthetic data introduces a measurable domain shift. The proposed framework provides a reproducible and scalable tool for surgical computer vision, supporting future research in data augmentation, domain adaptation, and simulation-based pretraining for robotic-assisted surgery. Data and code are available at https://github.com/EIDOSLAB/Sintetic-dataset-DaVinci.

Problem

• robotic surgery
• instrument segmentation
• synthetic dataset
• data scarcity
• surgical computer vision

Innovation

• synthetic dataset generation
• robotic surgery segmentation
• photorealistic simulation
• domain adaptation
• 3D animation pipeline