AnythingReality: Robust Online Gaussian Splatting SLAM for Open-Vocabulary VR Scene Exploration

2026-07-10

AI Summary

This work addresses the challenge of real-time, robust online 3D reconstruction and open-vocabulary interaction under real-world noisy data. The authors propose an incremental reconstruction framework that integrates ORB-SLAM3 for pose estimation with online 3D Gaussian splatting, coupled with a speech-driven vision-language model (VLM) and a VR rendering pipeline. This system achieves end-to-end immersive semantic exploration without requiring clean depth maps or external pose inputs. It enables voice-triggered open-vocabulary object recognition via the VLM, attaining 88% accuracy, and significantly improves reconstruction quality—demonstrating up to a 14.5% increase in PSNR and a 21.6% reduction in LPIPS—on both a custom dataset and TUM-RGBD, while maintaining high frame rates for real-time performance.

Abstract

We present a novel integrated architecture for robust online 3D Gaussian splatting, real-time VR exploration, and speech-driven Vision-Language-Model interaction. Unlike methods assuming clean depth or external poses, our system combines ORB-SLAM3-based pose estimation with online Gaussian reconstruction for noisy real-world data. A VR pipeline enables immersive exploration of incremental reconstructions; a semantic module transcribes voice commands, generates scene descriptions, and records points of interest. Against state-of-the-art online Gaussian splatting methods, we improve image quality on our dataset (+14.5% PSNR, +8.6% SSIM, -14.3% LPIPS) and TUM-RGBD (+11.7% PSNR, +7.8% SSIM, -21.6% LPIPS), with comparable or superior frame rates via quality-speed configurations. We achieve an 88% VLM object-recognition rate.

Problem

• Gaussian Splatting
• SLAM
• VR Exploration
• Open-Vocabulary
• Vision-Language Model

Innovation

• Online Gaussian Splatting
• SLAM
• VR Exploration
• Vision-Language Model
• Open-Vocabulary Interaction