MAD-OOD: A Deep Learning Cluster-Driven Framework for an Out-of-Distribution Malware Detection and Classification

2025-12-19

AI Summary

To address the out-of-distribution (OOD) detection challenge posed by high intra-family variability in malware, this paper proposes a two-stage clustering-driven framework. In Stage I, a class-conditional spherical decision boundary is constructed via Gaussian discriminant analysis, augmented with multi-center Z-score distance analysis—enabling statistically interpretable OOD detection without OOD supervision. In Stage II, predictions from clustering, fine-tuned embeddings, and supervised classification are fused to enhance accuracy on known families. Key contributions include: (i) the first use of spherical boundaries to model intra-family variation; (ii) zero reliance on OOD training data; and (iii) theoretical interpretability coupled with robust anomaly localization. Evaluated on 25 known families and diverse novel OOD variants, the method achieves an AUC of 0.911—significantly surpassing state-of-the-art baselines—while ensuring high scalability, interpretability, and statistical rigor.

Technology Category

• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Machine Learning: Clustering
• Search and Optimization: Distributed Search

Application Category

• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Out of distribution (OOD) detection remains a critical challenge in malware classification due to the substantial intra family variability introduced by polymorphic and metamorphic malware variants. Most existing deep learning based malware detectors rely on closed world assumptions and fail to adequately model this intra class variation, resulting in degraded performance when confronted with previously unseen malware families. This paper presents MADOOD, a novel two stage, cluster driven deep learning framework for robust OOD malware detection and classification. In the first stage, malware family embeddings are modeled using class conditional spherical decision boundaries derived from Gaussian Discriminant Analysis (GDA), enabling statistically grounded separation of indistribution and OOD samples without requiring OOD data during training. Z score based distance analysis across multiple class centroids is employed to reliably identify anomalous samples in the latent space. In the second stage, a deep neural network integrates cluster based predictions, refined embeddings, and supervised classifier outputs to enhance final classification accuracy. Extensive evaluations on benchmark malware datasets comprising 25 known families and multiple novel OOD variants demonstrate that MADOOD significantly outperforms state of the art OOD detection methods, achieving an AUC of up to 0.911 on unseen malware families. The proposed framework provides a scalable, interpretable, and statistically principled solution for real world malware detection and anomaly identification in evolving cybersecurity environments.

Problem

• Detects unseen malware families using cluster-driven deep learning
• Models intra-family variability without OOD data during training
• Enhances classification accuracy with integrated cluster and neural network predictions

Innovation

• Two-stage cluster-driven deep learning framework
• Gaussian Discriminant Analysis for OOD detection
• Integration of cluster predictions and classifier outputs