Communication Efficient Robotic Mixed Reality with Gaussian Splatting Cross-Layer Optimization

2025-08-12

AI Summary

To address the high communication overhead incurred by wireless transmission of high-resolution images in robotic mixed reality (RoboMR), this paper proposes a cross-layer optimization framework based on Gaussian lattice modeling. At the simulator side, it enables on-demand rendering of real-world views to reduce the need for transmitting high-frequency imagery. The framework jointly optimizes content-switching policies and power allocation to achieve tight co-design among communication, rendering, and control. It is the first to enable multi-robot, low-power, and dynamically robust RoboMR interaction under ultra-low communication overhead. A hybrid data modeling approach is introduced to enhance dynamic scene representation. Experiments on wheeled and legged robot platforms demonstrate a 10× reduction in communication overhead and over 10× decrease in computational complexity, while maintaining high-fidelity, real-time MR interaction.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Search and Optimization: Sampling/Simulation-based Search
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Realizing low-cost communication in robotic mixed reality (RoboMR) systems presents a challenge, due to the necessity of uploading high-resolution images through wireless channels. This paper proposes Gaussian splatting (GS) RoboMR (GSMR), which enables the simulator to opportunistically render a photo-realistic view from the robot's pose by calling ``memory'' from a GS model, thus reducing the need for excessive image uploads. However, the GS model may involve discrepancies compared to the actual environments. To this end, a GS cross-layer optimization (GSCLO) framework is further proposed, which jointly optimizes content switching (i.e., deciding whether to upload image or not) and power allocation (i.e., adjusting to content profiles) across different frames by minimizing a newly derived GSMR loss function. The GSCLO problem is addressed by an accelerated penalty optimization (APO) algorithm that reduces computational complexity by over $10$x compared to traditional branch-and-bound and search algorithms. Moreover, variants of GSCLO are presented to achieve robust, low-power, and multi-robot GSMR. Extensive experiments demonstrate that the proposed GSMR paradigm and GSCLO method achieve significant improvements over existing benchmarks on both wheeled and legged robots in terms of diverse metrics in various scenarios. For the first time, it is found that RoboMR can be achieved with ultra-low communication costs, and mixture of data is useful for enhancing GS performance in dynamic scenarios.

Problem

• Reducing communication costs in robotic mixed reality systems
• Optimizing content switching and power allocation across frames
• Enhancing Gaussian splatting performance in dynamic environments

Innovation

• Uses Gaussian Splatting for photo-realistic rendering
• Implements cross-layer optimization for content switching
• Reduces complexity with accelerated penalty algorithm