GS-Verse: Mesh-based Gaussian Splatting for Physics-aware Interaction in Virtual Reality

2025-10-13

AI Summary

Existing VR-based 3D physical manipulation methods rely on engineered pipelines and simplified geometric proxies (e.g., tetrahedral cages), compromising visual fidelity and mechanical realism. To address this, we propose GS-Verse—the first framework to explicitly embed 3D mesh structures into Gaussian lattices, enabling joint modeling of geometric detail and physically grounded behavior. GS-Verse integrates differentiable Gaussian rendering, mesh-lattice fusion, a decoupled physics engine interface, and real-time surface approximation to support photorealistic deformation, efficient user interaction, and cross-engine deployment. A user study demonstrates that GS-Verse significantly outperforms baseline methods across stretching, twisting, and jiggling tasks, achieving simultaneous improvements in manipulation accuracy, simulation stability, and visual quality.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Intelligent Robots: Manipulation
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

As the demand for immersive 3D content grows, the need for intuitive and efficient interaction methods becomes paramount. Current techniques for physically manipulating 3D content within Virtual Reality (VR) often face significant limitations, including reliance on engineering-intensive processes and simplified geometric representations, such as tetrahedral cages, which can compromise visual fidelity and physical accuracy. In this paper, we introduce our{} ( extbf{G}aussian extbf{S}platting for extbf{V}irtual extbf{E}nvironment extbf{R}endering and extbf{S}cene extbf{E}diting), a novel method designed to overcome these challenges by directly integrating an object's mesh with a Gaussian Splatting (GS) representation. Our approach enables more precise surface approximation, leading to highly realistic deformations and interactions. By leveraging existing 3D mesh assets, our{} facilitates seamless content reuse and simplifies the development workflow. Moreover, our system is designed to be physics-engine-agnostic, granting developers robust deployment flexibility. This versatile architecture delivers a highly realistic, adaptable, and intuitive approach to interactive 3D manipulation. We rigorously validate our method against the current state-of-the-art technique that couples VR with GS in a comparative user study involving 18 participants. Specifically, we demonstrate that our approach is statistically significantly better for physics-aware stretching manipulation and is also more consistent in other physics-based manipulations like twisting and shaking. Further evaluation across various interactions and scenes confirms that our method consistently delivers high and reliable performance, showing its potential as a plausible alternative to existing methods.

Problem

• Enabling precise physics-aware interaction with 3D content in Virtual Reality
• Overcoming limitations of simplified geometric representations in VR manipulation
• Integrating mesh assets with Gaussian Splatting for realistic deformations

Innovation

• Integrates object mesh with Gaussian Splatting representation
• Enables precise surface approximation for realistic deformations
• Physics-engine-agnostic design ensures deployment flexibility