A Multi-task Mixture of Experts Framework for Malware Classification, Packing Detection, and Family Attribution

2026-06-29

AI Summary

This work addresses the performance degradation of conventional single-model approaches in malware analysis, which stems from high heterogeneity, widespread packing, and diverse family distributions. To overcome these challenges, the authors propose a unified multi-task framework based on Mixture of Experts (MoE) that jointly performs benign/malicious classification, packer detection, and malware family identification. The framework employs a multi-gated MoE architecture to enable task-specific routing and expert specialization, integrating dual-modal inputs comprising high-dimensional EMBER features and raw byte sequences. Experimental results demonstrate that the proposed method significantly enhances generalization and robustness under both standard and adversarial conditions, achieving a composite detection rate of 0.9744 with a failure rate of only 2.56%, and exhibits exceptional resilience under distribution shifts.

Abstract

Malware classification remains a challenging problem due to its inherent heterogeneity, the presence of packed binaries, and the diverse distribution of malware families. Traditional single-model detection mechanisms often fail to generalize across such diverse data, leading to degraded performance, particularly on obfuscated and rare malware samples. In this work, we propose a unified multi-task malware analysis framework based on Mixture of Experts (MoE) architectures. The proposed system evaluates performance across two different input representations, i.e., high-dimensional EMBER feature sets and raw 1D byte arrays extracted from Portable Executable files. It simultaneously performs three critical tasks: malware family classification, packed versus unpacked detection, and malware versus benign identification. By decomposing the problem into specialized expert networks and employing adaptive gating mechanisms, the model enables effective task-specific learning while maintaining overall scalability. We investigate multiple architectural variants, including Homogeneous MoE, Heterogeneous MoE, and Multi-Gate MoE (MMoE). Performance is evaluated in both standard and adversarial settings using original and mutated samples. The obtained results demonstrate that the Multi-Gate MoE model achieves the best performance, reaching a combined detection rate of 0.9744 with only $2.56\%$ failure rate. Moreover, this configuration exhibits improved robustness under mutation-induced distribution shifts. Our findings highlight the effectiveness of expert specialization and task-specific routing in handling complex malware distributions, making the proposed framework a promising direction for scalable and resilient malware detection systems.

Problem

• malware classification
• packed binaries
• malware families
• heterogeneity
• obfuscated samples

Innovation

• Mixture of Experts
• Multi-task Learning
• Malware Classification
• Packing Detection
• Adversarial Robustness