SimPRIVE: a Simulation framework for Physical Robot Interaction with Virtual Environments

2025-04-30

AI Summary

To address the unpredictability, high testing costs, and safety risks associated with deploying machine learning and reinforcement learning algorithms on physical robots, this paper proposes a lightweight Vehicle-in-the-Loop (VIL) simulation framework. Built upon ROS 2 and Unreal Engine 5, it achieves, for the first time, real-time, bidirectional closed-loop synchronization between physical robots and high-fidelity virtual environments—enabling digital-twin motion replication, programmable dynamic obstacles, and full-stack algorithm safety verification. Key innovations include LiDAR-perception-driven control, a low-latency synchronization protocol, and scalable virtual modeling—balancing Hardware-in-the-Loop (HIL) fidelity with real-time rendering performance. Experimental validation on the AgileX Scout Mini robot demonstrates zero-collision operation of a reinforcement learning obstacle-avoidance agent; obstacle interaction accuracy exceeds 98% in a virtual office scenario; and end-to-end system latency remains below 80 ms.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Low Level & Physics-based Vision

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search

Abstract

The use of machine learning in cyber-physical systems has attracted the interest of both industry and academia. However, no general solution has yet been found against the unpredictable behavior of neural networks and reinforcement learning agents. Nevertheless, the improvements of photo-realistic simulators have paved the way towards extensive testing of complex algorithms in different virtual scenarios, which would be expensive and dangerous to implement in the real world. This paper presents SimPRIVE, a simulation framework for physical robot interaction with virtual environments, which operates as a vehicle-in-the-loop platform, rendering a virtual world while operating the vehicle in the real world. Using SimPRIVE, any physical mobile robot running on ROS 2 can easily be configured to move its digital twin in a virtual world built with the Unreal Engine 5 graphic engine, which can be populated with objects, people, or other vehicles with programmable behavior. SimPRIVE has been designed to accommodate custom or pre-built virtual worlds while being light-weight to contain execution times and allow fast rendering. Its main advantage lies in the possibility of testing complex algorithms on the full software and hardware stack while minimizing the risks and costs of a test campaign. The framework has been validated by testing a reinforcement learning agent trained for obstacle avoidance on an AgileX Scout Mini rover that navigates a virtual office environment where everyday objects and people are placed as obstacles. The physical rover moves with no collision in an indoor limited space, thanks to a LiDAR-based heuristic.

Problem

• Develops a simulation framework for physical robot-virtual environment interaction
• Enables safe testing of complex algorithms in virtual scenarios
• Integrates real-world robots with customizable virtual environments

Innovation

• SimPRIVE integrates physical robots with virtual environments
• Uses Unreal Engine 5 for realistic digital twin simulation
• Enables safe testing of reinforcement learning algorithms