CAN We Trust Your Results? A Cross-Dataset Study of Automotive IDS Evaluation

2026-06-29

AI Summary

This study addresses the lack of standardized evaluation protocols in existing automotive intrusion detection systems (IDS), which leads to performance assessments heavily dependent on specific datasets and poorly reflective of real-world generalization capabilities. To remedy this, the authors propose the first cross-dataset benchmarking framework, integrating seven publicly available CAN bus datasets under a unified preprocessing pipeline and consistent evaluation metrics. They systematically evaluate five representative IDS approaches—spanning statistical, machine learning, and deep learning paradigms—through comprehensive cross-dataset validation. The experiments reveal substantial performance fluctuations across datasets, with certain methods failing entirely in unseen environments, thereby exposing significant biases inherent in single-dataset evaluations. These findings underscore the critical need for cross-dataset validation and advocate for establishing standardized, reproducible evaluation paradigms in automotive security research.

Abstract

The increasing connectivity of modern vehicles has made securing in-vehicle communication networks a critical challenge. Intrusion Detection Systems (IDS) have been widely studied as a defense mechanism for detecting malicious activities on the Controller Area Network (CAN) bus. However, the evaluation of CAN IDS methods remains difficult due to inconsistencies in experimental setups and the lack of standardized benchmarking frameworks. As a result, reported performance often depends on dataset-specific characteristics and may not reflect how detection methods behave in different environments. This work introduces a benchmarking framework for consistent evaluation of CAN IDSs across multiple datasets. Using the proposed framework, we integrate seven publicly available CAN IDS datasets collected under different experimental conditions and perform cross-dataset evaluation of five conceptually different IDS approaches. Our results highlight how detection performance can vary significantly across datasets, demonstrating the importance of cross-dataset benchmarking for assessing the robustness and generalization capabilities of CAN IDS methods.

Problem

• CAN
• Intrusion Detection System
• cross-dataset evaluation
• benchmarking
• generalization

Innovation

• cross-dataset evaluation
• CAN IDS benchmarking
• intrusion detection robustness
• generalization capability
• standardized evaluation framework