CogRail: Benchmarking VLMs in Cognitive Intrusion Perception for Intelligent Railway Transportation Systems

2026-01-14

AI Summary

Existing railway intrusion detection systems struggle to effectively perceive potential risk objects due to a lack of comprehensive understanding of both spatial context and temporal dynamics. To address this limitation, this work proposes CogRail—the first multimodal benchmark tailored for cognitive-aware railway intrusion perception. CogRail introduces cognition-driven question-answering annotations to support spatiotemporal reasoning and develops a multi-task joint fine-tuning framework that simultaneously optimizes vision-language models for location awareness, motion prediction, and threat analysis. Experimental results demonstrate that general-purpose multimodal models exhibit limited performance on this task, whereas the proposed approach significantly improves both accuracy and interpretability, advancing the specialization of vision-language models for safety-critical applications.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Accurate and early perception of potential intrusion targets is essential for ensuring the safety of railway transportation systems. However, most existing systems focus narrowly on object classification within fixed visual scopes and apply rule-based heuristics to determine intrusion status, often overlooking targets that pose latent intrusion risks. Anticipating such risks requires the cognition of spatial context and temporal dynamics for the object of interest (OOI), which presents challenges for conventional visual models. To facilitate deep intrusion perception, we introduce a novel benchmark, CogRail, which integrates curated open-source datasets with cognitively driven question-answer annotations to support spatio-temporal reasoning and prediction. Building upon this benchmark, we conduct a systematic evaluation of state-of-the-art visual-language models (VLMs) using multimodal prompts to identify their strengths and limitations in this domain. Furthermore, we fine-tune VLMs for better performance and propose a joint fine-tuning framework that integrates three core tasks, position perception, movement prediction, and threat analysis, facilitating effective adaptation of general-purpose foundation models into specialized models tailored for cognitive intrusion perception. Extensive experiments reveal that current large-scale multimodal models struggle with the complex spatial-temporal reasoning required by the cognitive intrusion perception task, underscoring the limitations of existing foundation models in this safety-critical domain. In contrast, our proposed joint fine-tuning framework significantly enhances model performance by enabling targeted adaptation to domain-specific reasoning demands, highlighting the advantages of structured multi-task learning in improving both accuracy and interpretability. Code will be available at https://github.com/Hub-Tian/CogRail.

Problem

• cognitive intrusion perception
• railway transportation safety
• spatio-temporal reasoning
• visual-language models
• latent intrusion risk

Innovation

• cognitive intrusion perception
• visual-language models
• spatio-temporal reasoning
• joint fine-tuning
• railway safety benchmark