Digital Twin Generation from Visual Data: A Survey

2025-04-17

AI Summary

This work addresses the critical challenge of constructing high-fidelity, interactive digital twins from monocular video. We systematically survey state-of-the-art techniques—including 3D Gaussian splatting, generative inpainting, semantic segmentation, and multimodal foundation models—analyzing their synergistic mechanisms and capability boundaries in digital twin generation. For the first time, we unify their cross-scenario applications in robotic manipulation, media content creation, and building information modeling. We propose a comprehensive 3D evaluation framework assessing illumination robustness, occlusion recovery, and scalability, revealing fundamental bottlenecks in existing methods. Building on these insights, we introduce a novel lightweight dynamic reconstruction paradigm integrated with neural rendering, significantly improving real-time performance and generalization across diverse scenes. Our framework provides both theoretical foundations and a practical technical roadmap for deploying video-driven digital twins.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

This survey explores recent developments in generating digital twins from videos. Such digital twins can be used for robotics application, media content creation, or design and construction works. We analyze various approaches, including 3D Gaussian Splatting, generative in-painting, semantic segmentation, and foundation models highlighting their advantages and limitations. Additionally, we discuss challenges such as occlusions, lighting variations, and scalability, as well as potential future research directions. This survey aims to provide a comprehensive overview of state-of-the-art methodologies and their implications for real-world applications. Awesome list: https://github.com/ndrwmlnk/awesome-digital-twins

Problem

• Surveying digital twin generation from visual data
• Analyzing methods for robotics and content creation
• Addressing challenges like occlusions and scalability

Innovation

• Generates digital twins from videos
• Uses 3D Gaussian Splatting techniques
• Applies semantic segmentation methods