EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers

2025-06-05

AI Summary

Existing public malware datasets suffer from three key limitations: single-platform coverage, coarse-grained labeling, and absence of real-world evasion samples—hindering robustness evaluation. To address these, we introduce MalBench, the first open-source benchmark supporting multi-task learning, cross-format analysis, and real-world false-negative samples. It encompasses over 3.2 million samples across six file formats and enables seven tasks—including malware detection, family classification, and behavioral identification. Our contributions are threefold: (1) a novel “Challenge Set” of evasion samples derived from commercial antivirus false negatives; (2) EMBER v3 features integrating fine-grained static and dynamic attributes; and (3) a multi-label annotation schema with an efficient metadata management framework. MalBench fully releases sample hashes, metadata, feature vectors, and multidimensional labels, substantially enhancing model reproducibility and comprehensive evaluation. It has already facilitated breakthroughs in detection accuracy and generalization for multiple state-of-the-art methods.

Technology Category

• Machine Learning: Evaluation and Analysis
• Application Domains: Misinformation & Fake News
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

A lack of accessible data has historically restricted malware analysis research, and practitioners have relied heavily on datasets provided by industry sources to advance. Existing public datasets are limited by narrow scope - most include files targeting a single platform, have labels supporting just one type of malware classification task, and make no effort to capture the evasive files that make malware detection difficult in practice. We present EMBER2024, a new dataset that enables holistic evaluation of malware classifiers. Created in collaboration with the authors of EMBER2017 and EMBER2018, the EMBER2024 dataset includes hashes, metadata, feature vectors, and labels for more than 3.2 million files from six file formats. Our dataset supports the training and evaluation of machine learning models on seven malware classification tasks, including malware detection, malware family classification, and malware behavior identification. EMBER2024 is the first to include a collection of malicious files that initially went undetected by a set of antivirus products, creating a"challenge"set to assess classifier performance against evasive malware. This work also introduces EMBER feature version 3, with added support for several new feature types. We are releasing the EMBER2024 dataset to promote reproducibility and empower researchers in the pursuit of new malware research topics.

Problem

• Lack of comprehensive malware datasets for holistic evaluation
• Existing datasets limited to single platforms and tasks
• Need for challenge set with evasive malware samples

Innovation

• EMBER2024 dataset supports six file formats
• Includes challenge set for evasive malware
• Introduces EMBER feature version 3