GSAC: Leveraging Gaussian Splatting for Photorealistic Avatar Creation with Unity Integration

2025-04-17

AI Summary

Existing virtual human modeling approaches suffer from high costs, long production cycles, and poor real-time performance: manual modeling relies heavily on skilled artists, while automated methods severely compromise facial expression fidelity, rendering efficiency, and robustness to in-the-wild monocular video input. This paper introduces the first end-to-end 3D Gaussian Splatting (3DGS)-based framework for virtual human generation from unconstrained monocular video. It integrates customized video preprocessing, Gaussian splatting optimization, and dynamic facial modeling to achieve markerless, high-fidelity expression reconstruction. Furthermore, it pioneers seamless integration of the 3DGS-driven model into the Unity engine, enabling rigged avatar binding and interactive editing. The system achieves real-time rendering at >60 FPS with photorealistic quality on standard hardware, reducing production cost by over 80%. Experimental validation confirms cross-platform deployability and industrial viability.

Technology Category

• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Computer Vision: 3D Computer Vision
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures

Abstract

Photorealistic avatars have become essential for immersive applications in virtual reality (VR) and augmented reality (AR), enabling lifelike interactions in areas such as training simulations, telemedicine, and virtual collaboration. These avatars bridge the gap between the physical and digital worlds, improving the user experience through realistic human representation. However, existing avatar creation techniques face significant challenges, including high costs, long creation times, and limited utility in virtual applications. Manual methods, such as MetaHuman, require extensive time and expertise, while automatic approaches, such as NeRF-based pipelines often lack efficiency, detailed facial expression fidelity, and are unable to be rendered at a speed sufficent for real-time applications. By involving several cutting-edge modern techniques, we introduce an end-to-end 3D Gaussian Splatting (3DGS) avatar creation pipeline that leverages monocular video input to create a scalable and efficient photorealistic avatar directly compatible with the Unity game engine. Our pipeline incorporates a novel Gaussian splatting technique with customized preprocessing that enables the user of"in the wild"monocular video capture, detailed facial expression reconstruction and embedding within a fully rigged avatar model. Additionally, we present a Unity-integrated Gaussian Splatting Avatar Editor, offering a user-friendly environment for VR/AR application development. Experimental results validate the effectiveness of our preprocessing pipeline in standardizing custom data for 3DGS training and demonstrate the versatility of Gaussian avatars in Unity, highlighting the scalability and practicality of our approach.

Problem

• High cost and time in photorealistic avatar creation
• Limited efficiency and fidelity in existing automatic methods
• Challenges in real-time rendering for VR/AR applications

Innovation

• Uses Gaussian Splatting for photorealistic avatars
• Integrates with Unity for real-time applications
• Leverages monocular video for efficient creation