Enhancing Decision-Making in Windows PE Malware Classification During Dataset Shifts with Uncertainty Estimation

2025-12-20

AI Summary

Windows PE malware classifiers suffer from reduced reliability and increased misclassification risk under data distribution shifts—e.g., when packed samples dominate the UCSB dataset. Method: We propose an uncertainty-aware robust detection framework built upon LightGBM, integrating predictive uncertainty modeling—formulated as ensemble non-conformity—as a core component of inductive conformal prediction (ICP). We further introduce a novel threshold optimization strategy that dynamically balances the false acceptance rate (FAR) and correct acceptance rate (CAR). Contribution/Results: This is the first work to embed ensemble-based uncertainty into ICP for PE malware classification. Our framework significantly improves out-of-distribution generalization: on the UCSB dataset, FAR drops to 16%, representing a ~30% reduction over the state-of-the-art, while maintaining competitive CAR. The approach substantially enhances robustness against distributional shifts.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Security and Privacy: Large-scale security measurements

Abstract

Artificial intelligence techniques have achieved strong performance in classifying Windows Portable Executable (PE) malware, but their reliability often degrades under dataset shifts, leading to misclassifications with severe security consequences. To address this, we enhance an existing LightGBM (LGBM) malware detector by integrating Neural Networks (NN), PriorNet, and Neural Network Ensembles, evaluated across three benchmark datasets: EMBER, BODMAS, and UCSB. The UCSB dataset, composed mainly of packed malware, introduces a substantial distributional shift relative to EMBER and BODMAS, making it a challenging testbed for robustness. We study uncertainty-aware decision strategies, including probability thresholding, PriorNet, ensemble-derived estimates, and Inductive Conformal Evaluation (ICE). Our main contribution is the use of ensemble-based uncertainty estimates as Non-Conformity Measures within ICE, combined with a novel threshold optimisation method. On the UCSB dataset, where the shift is most severe, the state-of-the-art probability-based ICE (SOTA) yields an incorrect acceptance rate (IA%) of 22.8%. In contrast, our method reduces this to 16% a relative reduction of about 30% while maintaining competitive correct acceptance rates (CA%). These results demonstrate that integrating ensemble-based uncertainty with conformal prediction provides a more reliable safeguard against misclassifications under extreme dataset shifts, particularly in the presence of packed malware, thereby offering practical benefits for real-world security operations.

Problem

• Enhance malware classification reliability under dataset shifts
• Reduce misclassification rates for packed malware using uncertainty estimation
• Integrate ensemble methods with conformal prediction for robust decision-making

Innovation

• Integrating ensemble-based uncertainty with conformal prediction
• Using ensemble uncertainty as Non-Conformity Measures in ICE
• Novel threshold optimisation method for improved reliability