MIRAGE: Enabling Real-Time Automotive Mediated Reality

2026-01-27

AI Summary

This work addresses the absence of real-world deployable, real-time Automotive Mediated Reality (AMR) systems in existing research, which has hindered empirical evaluation of their impact on driving safety. To bridge the gap between simulation and real-world assessment, the authors present MIRAGE—an open-source framework that, for the first time, enables in-vehicle implementation of 15 distinct AMR effects spanning augmentation, attenuation, and modification of visual content. Built upon state-of-the-art models for object detection, semantic segmentation, depth estimation, and image inpainting, MIRAGE supports real-time visual interventions on diverse traffic entities. In an on-road user study involving nine domain experts, participants consistently recognized AMR’s potential to enhance situational awareness and mitigate driver distraction, while also identifying concrete use cases and directions for future refinement.

Technology Category

• Computer Vision: Multi-modal Vision
• Application Domains: Mobility, Driving & Flight
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health

Abstract

Traffic is inherently dangerous, with around 1.19 million fatalities annually. Automotive Mediated Reality (AMR) can enhance driving safety by overlaying critical information (e.g., outlines, icons, text) on key objects to improve awareness, altering objects'appearance to simplify traffic situations, and diminishing their appearance to minimize distractions. However, real-world AMR evaluation remains limited due to technical challenges. To fill this sim-to-real gap, we present MIRAGE, an open-source tool that enables real-time AMR in real vehicles. MIRAGE implements 15 effects across the AMR spectrum of augmented, diminished, and modified reality using state-of-the-art computational models for object detection and segmentation, depth estimation, and inpainting. In an on-road expert user study (N=9) of MIRAGE, participants enjoyed the AMR experience while pointing out technical limitations and identifying use cases for AMR. We discuss these results in relation to prior work and outline implications for AMR ethics and interaction design.

Problem

• Automotive Mediated Reality
• real-time
• sim-to-real gap
• driving safety
• in-vehicle evaluation

Innovation

• Automotive Mediated Reality
• real-time AR/VR
• object detection and segmentation
• depth estimation
• inpainting