A Mixed-Reality Testbed for Autonomous Vehicles

2026-06-17

AI Summary

This work addresses the validation gap between simulation and real-world deployment of autonomous driving algorithms, particularly the lack of efficient, high-fidelity testing platforms for safety-critical scenarios. To bridge this gap, the authors propose a mixed-reality hardware-in-the-loop testing framework that seamlessly integrates physical mobile robots with high-fidelity virtual environments, enabling multimodal sensing, vehicle-to-everything (V2X) communication, and large-scale multi-agent collaboration. A key innovation is the coexistence of physical and virtual agents within a unified architecture, coupled with an online learning controller based on control barrier functions (CBFs) that establishes an integrated perception-planning-control safety assurance mechanism. Experimental results demonstrate that the platform significantly enhances the reliability and efficiency of sim-to-real transfer and validates its effectiveness across diverse safety-critical scenarios.

Abstract

We propose a mixed-reality, hardware-in-the-loop (HIL) testbed for autonomous vehicles that seamlessly integrates a physical testbed of mobile robots with a high-fidelity simulation environment. The virtual simulation enables the creation of diverse, safety-critical driving scenarios to validate state-of-the-art perception, planning, and control algorithms, while augmenting simulations with physical robots equipped with multimodal sensors in photorealistic virtual environments further facilitating rigorous validation. Our testbed also features vehicular connectivity using wireless communication and can accommodate a large number of agents through the combination of physical robots and virtual simulated agents, supporting research on multi-agent systems including Connected and Autonomous Vehicles (CAVs). Finally, we present a safety-guaranteed framework combining perception, planning and a novel online learning-based controller using Control Barrier Functions (CBFs) for CAVs. Experiments using the proposed framework are used to validate and demonstrate the key functionalities and the overall utility of the testbed to bridge the gap between simulation and real-world hardware deployment.

Problem

• Autonomous Vehicles
• Mixed-Reality
• Hardware-in-the-Loop
• Safety-Critical Scenarios
• Connected and Autonomous Vehicles

Innovation

• Mixed-Reality
• Hardware-in-the-Loop
• Control Barrier Functions
• Connected and Autonomous Vehicles
• Online Learning-based Control