Revisiting Static Feature-Based Android Malware Detection

2024-09-11

arXiv.org

AI Summary

Existing static-feature-based machine learning approaches for Android malware detection achieve high efficiency and scalability but suffer from severe reproducibility issues—including dataset reuse, insufficient hyperparameter tuning, and uncontrolled random seeds—leading to inflated and unreliable performance estimates. Method: We conduct a systematic evaluation of six mainstream models (e.g., RF, SVM), quantifying the impact of data bias and flawed experimental design; rigorously tune all models; and propose standardized protocols for dataset construction and evaluation. Contribution/Results: We find that well-tuned simple baselines frequently outperform complex models. To address reproducibility gaps, we release a lightweight, extensible, open-source experimental framework—including full code, configurations, and documentation. Empirical results demonstrate that properly configured baselines achieve state-of-the-art accuracy across multiple benchmarks, significantly enhancing fairness, transparency, and trustworthiness in comparative model evaluation.

Technology Category

• Machine Learning: Evaluation and Analysis
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The increasing reliance on machine learning (ML) in computer security, particularly for malware classification, has driven significant advancements. However, the replicability and reproducibility of these results are often overlooked, leading to challenges in verifying research findings. This paper highlights critical pitfalls that undermine the validity of ML research in Android malware detection, focusing on dataset and methodological issues. We comprehensively analyze Android malware detection using two datasets and assess offline and continual learning settings with six widely used ML models. Our study reveals that when properly tuned, simpler baseline methods can often outperform more complex models. To address reproducibility challenges, we propose solutions for improving datasets and methodological practices, enabling fairer model comparisons. Additionally, we open-source our code to facilitate malware analysis, making it extensible for new models and datasets. Our paper aims to support future research in Android malware detection and other security domains, enhancing the reliability and reproducibility of published results.

Problem

• Addresses reproducibility issues in Android malware detection systems
• Evaluates machine learning models under rigorous selection methodology
• Compares simple versus complex models on deduplicated datasets

Innovation

• Uses rigorous model selection and evaluation methodology
• Evaluates simpler tree-based models outperforming complex neural networks
• Provides open-source extensible codebase for reproducibility