Demystifying Feature Engineering in Malware Analysis of API Call Sequences

2025-12-01

AI Summary

This study addresses the feature selection challenge for API call sequences in malware detection. We systematically compare domain-knowledge-driven and NLP-inspired feature engineering approaches across CNN, LSTM, and Transformer models. Results show that domain-informed features—grounded in runtime semantics such as handles and virtual addresses—consistently outperform NLP-style statistical features (e.g., n-grams, TF-IDF) in accuracy, F1-score, and few-shot generalization. Moreover, ablation and interpretability analyses reveal that models predominantly rely on highly dynamic, low-interpretable execution-state features, challenging the conventional assumption that API frequency or sequential order alone constitutes meaningful semantic signal. Our work establishes a reproducible feature design paradigm for API sequence modeling and introduces a critical framework for re-examining model interpretability in system-call-based malware analysis.

Technology Category

• Machine Learning: Feature Construction/Reformulation
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Machine learning (ML) has been widely used to analyze API call sequences in malware analysis, which typically requires the expertise of domain specialists to extract relevant features from raw data. The extracted features play a critical role in malware analysis. Traditional feature extraction is based on human domain knowledge, while there is a trend of using natural language processing (NLP) for automatic feature extraction. This raises a question: how do we effectively select features for malware analysis based on API call sequences? To answer it, this paper presents a comprehensive study of investigating the impact of feature engineering upon malware classification.We first conducted a comparative performance evaluation under three models, Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and Transformer, with respect to knowledge-based and NLP-based feature engineering methods. We observed that models with knowledge-based feature engineering inputs generally outperform those using NLP-based across all metrics, especially under smaller sample sizes. Then we analyzed a complete set of data features from API call sequences, our analysis reveals that models often focus on features such as handles and virtual addresses, which vary across executions and are difficult for human analysts to interpret.

Problem

• Investigating feature engineering impact on malware classification
• Comparing knowledge-based vs NLP-based feature extraction methods
• Analyzing model focus on variable and hard-to-interpret features

Innovation

• Using knowledge-based feature engineering for malware classification
• Comparing CNN, LSTM, Transformer models with API call sequences
• Focusing on handles and virtual addresses as key features