Metric-Guided Synthetic Image Data Rendering for Deep Learning compatible with Agentic AI

2026-07-14

AI Summary

This work addresses the high cost of annotating real-world data and the lack of quantitative guidance for mitigating domain gaps between synthetic and real images in scientific vision tasks. To this end, the authors propose a programmable 3D rendering framework that systematically enhances the realism, diversity, and scale of synthetic data by incorporating quantitative metrics—such as gradient similarity and zero-shot detection performance—and encapsulates the rendering pipeline as an agent skill for automated parameter optimization. This approach represents the first integration of quantitatively guided synthetic data refinement into an agent-based framework, substantially improving model visual perception: it boosts zero-shot object detection performance and further refines small-object detection when trained on mixed real-synthetic datasets. The implementation leverages the authors’ custom Python toolkit, GraNatPy, which includes the SynthClaw agent.

Abstract

Deep learning computer vision for scientific applications requires collecting and annotating large datasets in a laborious, expensive and error-prone process. Synthetic data generation through 3D modelling and rendering may simplify this process and increase the accuracy of annotations by generating them programmatically. However, minimising the domain gap between real and synthetic images visually is subjective and lacks systematic quantitative guidance. We present GraNatPy, a Python package with metrics to guide improvement of the rendered scene. We show that quantifiable increase in realism, diversity and size of rendered dataset correlates with improved visual perception of the scene and higher zero-shot performance of an object detection model. Furthermore, we demonstrated using photographs of virological plaque assays that gradient similarity affects performance on small object detection, which can be improved by mixing real and synthetic data. Finally, we turn procedural data rendering into an agentic skill (SynthClaw) to automate the procedural parameter optimisation.

Problem

• synthetic data
• domain gap
• computer vision
• deep learning
• quantitative metrics

Innovation

• metric-guided rendering
• synthetic data generation
• domain gap reduction
• agentic AI
• zero-shot object detection