Benchmarking Android Malware Detection: Rethinking the Role of Traditional and Deep Learning Models

2025-02-20

AI Summary

This paper addresses the lack of systematic and rigorous comparative evaluation between deep learning (DL) and traditional machine learning (ML) models in Android malware detection. We establish a unified benchmarking framework across four diverse datasets, systematically comparing traditional models—including Random Forest and CatBoost—with DL models such as CapsGNN, BERT, and ExcelFormer. Results show that CatBoost achieves an average F1-score of 98.7% on both static and dynamic features, significantly outperforming most DL approaches; traditional models are frequently underestimated, yet demonstrate superior efficiency and generalization for practical deployment. To promote standardized evaluation, we propose the “mandatory multi-baseline traditional model” benchmarking paradigm. Furthermore, we publicly release a high-quality, manually annotated dataset comprising over 120,000 samples, along with full experimental code.

Technology Category

• Machine Learning: Evaluation and Analysis
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Computer Vision: Large Vision Models

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Android malware detection has been extensively studied using both traditional machine learning (ML) and deep learning (DL) approaches. While many state-of-the-art detection models, particularly those based on DL, claim superior performance, they often rely on limited comparisons, lacking comprehensive benchmarking against traditional ML models across diverse datasets. This raises concerns about the robustness of DL-based approaches' performance and the potential oversight of simpler, more efficient ML models. In this paper, we conduct a systematic evaluation of Android malware detection models across four datasets: three recently published, publicly available datasets and a large-scale dataset we systematically collected. We implement a range of traditional ML models, including Random Forests (RF) and CatBoost, alongside advanced DL models such as Capsule Graph Neural Networks (CapsGNN), BERT-based models, and ExcelFormer based models. Our results reveal that while advanced DL models can achieve strong performance, they are often compared against an insufficient number of traditional ML baselines. In many cases, simpler and more computationally efficient ML models achieve comparable or even superior performance. These findings highlight the need for rigorous benchmarking in Android malware detection research. We encourage future studies to conduct more comprehensive benchmarking comparisons between traditional and advanced models to ensure a more accurate assessment of detection capabilities. To facilitate further research, we provide access to our dataset, including app IDs, hash values, and labels.

Problem

• Comprehensive benchmarking of Android malware detection models
• Comparison of traditional ML and deep learning approaches
• Evaluating performance across diverse datasets

Innovation

• Comprehensive benchmarking of detection models
• Comparison of traditional ML and DL models
• Public dataset provision for further research