GaussLite: Online Task-Conditioned 3D Gaussian Splatting for Real-Time Robotic Mapping

2026-06-29

AI Summary

This work addresses the limitation of conventional 3D Gaussian splatting methods, which allocate resources uniformly and thus fail to meet the high-fidelity reconstruction demands of robotic tasks in specific regions. We propose the first task-driven, online 3D Gaussian splatting mapping system that leverages a large language model to interpret natural language instructions, integrating open-vocabulary object detection with pixel-level correspondence masks to dynamically modulate Gaussian density for on-demand resource allocation. The system supports real-time multi-agent map fusion and sharing, achieving a 2.72 dB and 2.23 dB PSNR improvement in task-relevant regions under identical Gaussian budgets on Replica and real-world scenes, respectively. Multi-agent fusion transmits only 7.08% of the data while yielding a 3.42 dB PSNR gain over stitching-based approaches, and maintains real-time performance at 4 Hz even on resource-constrained devices.

Abstract

Existing 3D Gaussian Splatting (3DGS) systems distribute representation capacity uniformly across a scene, ignoring the fact that many downstream robotic tasks engage only a fraction of the reconstructed geometry. This causes valuable onboard compute to be allocated towards optimizing irrelevant parts of the scene, either limiting online capacity or under-optimizing the most relevant parts of the scene. We introduce GaussLite, a task-driven 3DGS mapping system that conditions its representation density on a natural-language task specification. Given a posed RGB-D stream and a task such as "prepare to pick up the object on the desk," GaussLite uses a one-shot LLM parser to extract target and anchor objects, which are grounded per-frame by an open-vocabulary detector and segmented to produce per-pixel relevance masks in real time. The mapper allocates seeding density, gradient flow and scaling by task relevance. At matched Gaussian budget and real-time mapping at 4 Hz on resource-constrained hardware, GaussLite outperforms baselines on ROI PSNR on the Replica Dataset by an average +2.72 dB and on a real-hardware demonstration in indoor and outdoor settings by +2.23 dB. We further show that two task-specialized agents' maps can be fused into a single shared map via per-voxel voting on active-optimization counts in real time, outperforming concatenation by +3.42 dB while only sharing an average 7.08% of the map.

Problem

• 3D Gaussian Splatting
• task-conditioned mapping
• robotic mapping
• representation efficiency
• real-time 3D reconstruction

Innovation

• task-conditioned
• 3D Gaussian Splatting
• real-time mapping
• language-guided perception
• multi-agent map fusion