Applying Graph Analysis for Unsupervised Fast Malware Fingerprinting

2025-10-07

AI Summary

Facing the deluge of novel malware samples daily, manual analysis is infeasible, necessitating efficient and scalable unsupervised semantic clustering methods. This paper proposes TrapNet, a static-analysis-based malware fingerprinting and clustering framework. Its core innovation is FloatHash—a numeric fuzzy hashing technique tailored to assembly instruction sequences—that generates compact real-valued vectors via PCA dimensionality reduction; these vectors are used to construct a weighted similarity graph, enabling high-purity family clustering through community detection. Experiments on large-scale datasets demonstrate that TrapNet significantly outperforms state-of-the-art methods, achieving superior coverage (>92%), +8.3% higher clustering purity, and substantially reduced computational overhead. TrapNet enables fully automated, scalable, and semantically meaningful malware grouping.

Technology Category

• Machine Learning: Clustering
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

Malware proliferation is increasing at a tremendous rate, with hundreds of thousands of new samples identified daily. Manual investigation of such a vast amount of malware is an unrealistic, time-consuming, and overwhelming task. To cope with this volume, there is a clear need to develop specialized techniques and efficient tools for preliminary filtering that can group malware based on semantic similarity. In this paper, we propose TrapNet, a novel, scalable, and unsupervised framework for malware fingerprinting and grouping. TrapNet employs graph community detection techniques for malware fingerprinting and family attribution based on static analysis, as follows: (1) TrapNet detects packed binaries and unpacks them using known generic packer tools. (2) From each malware sample, it generates a digest that captures the underlying semantics. Since the digest must be dense, efficient, and suitable for similarity checking, we designed FloatHash (FH), a novel numerical fuzzy hashing technique that produces a short real-valued vector summarizing the underlying assembly items and their order. FH is based on applying Principal Component Analysis (PCA) to ordered assembly items (e.g., opcodes, function calls) extracted from the malware's assembly code. (3) Representing malware with short numerical vectors enables high-performance, large-scale similarity computation, which allows TrapNet to build a malware similarity network. (4) Finally, TrapNet employs state-of-the-art community detection algorithms to identify dense communities, which represent groups of malware with similar semantics. Our extensive evaluation of TrapNet demonstrates its effectiveness in terms of the coverage and purity of the detected communities, while also highlighting its runtime efficiency, which outperforms other state-of-the-art solutions.

Problem

• Developing unsupervised framework for malware fingerprinting and grouping
• Creating efficient similarity network for large-scale malware analysis
• Applying graph community detection to identify semantic malware families

Innovation

• Unpacks malware using generic packer tools
• Generates fuzzy hash via PCA on assembly code
• Builds similarity network for community detection