Multi-modal Rail Crossing Safety Analysis

2026-07-01

AI Summary

This study addresses railway crossing safety assessment by proposing a multimodal AI approach that integrates visual imagery with structured accident data. For the first time, a compact vision-language model—fine-tuned via routing—is combined with historical accident records to jointly model safety rating regression and risk classification through multimodal fusion, aligning predictions with Federal Railroad Administration (FRA) standards and expert judgments. Experimental results demonstrate that the method achieves a macro F1-score of 0.757 in distinguishing high- and low-risk crossings, estimates FRA scores with an RMSE of 0.078 (Pearson correlation coefficient: 0.492), and yields qualitative assessments consistent with domain experts. These findings validate the effectiveness and innovation of the proposed multimodal learning paradigm in real-world safety evaluation scenarios.

Abstract

Given one or more images of a railway crossing, can we leverage visual cues that allow us to robustly estimate how safe it is? Can we improve our ability to do so by introducing structured data (such as official accident reports) about the accident history of that crossing into our models? In this work, we explore how to best answer those questions towards building an AI system that can ingest multi-modal data for railway crossings and provide safety assessment and scores that align with expert opinion and with safety scoring used by the Federal Railroad Administration (FRA). To that end, we propose a proof-of-concept pipeline that delivers on that goal, while at the same time exploring and tackling a number of critical research challenges that pertain to different parts of the pipeline, from data preparation to different learning paradigms that can allow us to realize such a system. Indicatively, our proposed system identifies HIGH-RISK and LOW-RISK crossings with a macro F1 score of 0.757 and estimates FRA-based safety scores with an RMSE of 0.078 and correlation of 0.492 using a routed fine-tuned compact VLM pipeline, while producing qualitative results that align with domain-expert assessment.

Problem

• railway crossing safety
• multi-modal data
• safety assessment
• visual cues
• accident history

Innovation

• multi-modal learning
• vision-language model
• railway crossing safety
• structured data integration
• FRA safety scoring