🤖 AI Summary
This study addresses the low efficiency of incident classification in aviation safety investigations. We establish the first systematic evaluation framework to assess five mainstream supervised learning models—SVM, logistic regression, random forest, XGBoost, and KNN—on a binary classification task distinguishing “incidents” from “serious incidents.” Our empirical analysis reveals, for the first time, that SMOTE significantly degrades classification performance, indicating its unsuitability for this domain. Across 100 repeated experiments, random forest achieves superior performance (accuracy: 0.77; F1-score: 0.78; Matthews correlation coefficient: 0.51). We further develop and deploy an interactive web application, now integrated into operational safety analysis workflows. The core contributions are: (1) a robust, empirically validated model selection paradigm for aviation safety text classification; and (2) domain-specific data preprocessing guidelines, notably advising against SMOTE use in this context.
📝 Abstract
This paper describes a practical approach of using supervised machine learning (ML) models to assist safety investigators to classify aviation occurrences into either incident or serious incident categories. Our implementation currently deployed as a ML web application is trained on a labelled dataset derived from publicly available aviation investigation reports. A selection of five supervised learning models (Support Vector Machine, Logistic Regression, Random Forest Classifier, XGBoost and K-Nearest Neighbors) were evaluated. This paper showed the best performing ML algorithm was the Random Forest Classifier with accuracy = 0.77, F1 Score = 0.78 and MCC = 0.51 (average of 100 sample runs). The study had also explored the effect of applying Synthetic Minority Over-sampling Technique (SMOTE) to the imbalanced dataset, and the overall observation ranged from no significant effect to substantial degradation in performance for some of the models after the SMOTE adjustment.