OSDaR-AR: Enhancing Railway Perception Datasets via Multi-modal Augmented Reality

2026-02-26

AI Summary

This work addresses the scarcity of high-quality, real-world annotated data that hinders the development of safety-critical railway intelligent perception systems, particularly for obstacle detection. To bridge this gap, the authors propose a multimodal augmented reality framework that seamlessly integrates LiDAR point clouds, INS/GNSS positioning, and image segmentation to embed high-fidelity virtual objects into real railway video sequences with precise spatiotemporal consistency. A key innovation is the introduction of a segmentation-based INS/GNSS refinement strategy, which significantly enhances the realism of the synthesized data. The study also constructs and publicly releases OSDaR-AR, the first augmented reality dataset tailored for railway perception, effectively narrowing the sim-to-real gap and providing a critical data foundation for advancing next-generation railway perception systems.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health
• Security and Privacy: Data transparency and provenance

Abstract

Although deep learning has significantly advanced the perception capabilities of intelligent transportation systems, railway applications continue to suffer from a scarcity of high-quality, annotated data for safety-critical tasks like obstacle detection. While photorealistic simulators offer a solution, they often struggle with the ``sim-to-real" gap; conversely, simple image-masking techniques lack the spatio-temporal coherence required to obtain augmented single- and multi-frame scenes with the correct appearance and dimensions. This paper introduces a multi-modal augmented reality framework designed to bridge this gap by integrating photorealistic virtual objects into real-world railway sequences from the OSDaR23 dataset. Utilizing Unreal Engine 5 features, our pipeline leverages LiDAR point-clouds and INS/GNSS data to ensure accurate object placement and temporal stability across RGB frames. This paper also proposes a segmentation-based refinement strategy for INS/GNSS data to significantly improve the realism of the augmented sequences, as confirmed by the comparative study presented in the paper. Carefully designed augmented sequences are collected to produce OSDaR-AR, a public dataset designed to support the development of next-generation railway perception systems. The dataset is available at the following page: https://syndra.retis.santannapisa.it/osdarar.html

Problem

• railway perception
• data scarcity
• sim-to-real gap
• spatio-temporal coherence
• obstacle detection

Innovation

• multi-modal augmented reality
• LiDAR-INS/GNSS fusion
• photorealistic augmentation
• railway perception dataset
• temporal coherence