System Calls for Malware Detection and Classification: Methodologies and Applications

2025-06-02

AI Summary

To address the growing challenge of detecting increasingly stealthy and sophisticated malware, this paper proposes a cross-platform (Windows/Linux/Android) malware detection and classification framework grounded in system call and API call behavior modeling. The method integrates static analysis, dynamic sandbox execution, and multi-paradigm behavioral modeling to construct a unified cross-platform system call abstraction layer. It introduces the first systematic integration of LSTM-based sequence modeling, SVM classification, statistical feature engineering, and interpretable anomaly detection—collectively forming a robust, evasion-resistant system call behavioral fingerprinting system. Experiments demonstrate classification accuracy exceeding 98% across mainstream platforms, significantly improving zero-day malware detection rates, reducing false positives, and effectively countering advanced evasion techniques such as API call obfuscation and delayed loading.

Technology Category

• Machine Learning: Evaluation and Analysis
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Computer Vision: Object Detection & Categorization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

As malware continues to become more complex and harder to detect, Malware Analysis needs to continue to evolve to stay one step ahead. One promising key area approach focuses on using system calls and API Calls, the core communication between user applications and the operating system and their kernels. These calls provide valuable insight into how software or programs behaves, making them an useful tool for spotting suspicious or harmful activity of programs and software. This chapter takes a deep down look at how system calls are used in malware detection and classification, covering techniques like static and dynamic analysis, as well as sandboxing. By combining these methods with advanced techniques like machine learning, statistical analysis, and anomaly detection, researchers can analyze system call patterns to tell the difference between normal and malicious behavior. The chapter also explores how these techniques are applied across different systems, including Windows, Linux, and Android, while also looking at the ways sophisticated malware tries to evade detection.

Problem

• Detecting complex malware using system call analysis
• Classifying malware behavior via static and dynamic techniques
• Evading detection across Windows, Linux, and Android systems

Innovation

• Uses system calls for malware detection
• Combines static and dynamic analysis techniques
• Applies machine learning to call patterns