RINSER: Accurate API Prediction Using Masked Language Models

2025-09-05

AI Summary

Malware frequently employs API obfuscation to impede behavioral analysis, resulting in inefficient manual inspection. To address this, we propose an automated API function name prediction method for Windows PE binaries—including x86 and stripped samples. Our approach introduces the novel concept of “API code fingerprints,” jointly modeling assembly instruction sequences and leveraging BERT-based masked language modeling. Evaluated on a large-scale dataset of 4.7 million samples, the method achieves 85.77% top-1 accuracy. It demonstrates strong adversarial robustness, outperforming state-of-the-art baselines by over 20% under obfuscation attacks. Crucially, our method successfully identifies 65 stealthy C2 communication and espionage-related APIs missed by leading commercial tools. This significantly enhances both rapid malware classification and in-depth behavioral analysis, enabling more effective detection of evasive threats.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Malware authors commonly use obfuscation to hide API identities in binary files, making analysis difficult and time-consuming for a human expert to understand the behavior and intent of the program. Automatic API prediction tools are necessary to efficiently analyze unknown binaries, facilitating rapid malware triage while reducing the workload on human analysts. In this paper, we present RINSER (AccuRate API predictioN using maSked languagE model leaRning), an automated framework for predicting Windows API (WinAPI) function names. RINSER introduces the novel concept of API codeprints, a set of API-relevant assembly instructions, and supports x86 PE binaries. RINSER relies on BERT's masked language model (LM) to predict API names at scale, achieving 85.77% accuracy for normal binaries and 82.88% accuracy for stripped binaries. We evaluate RINSER on a large dataset of 4.7M API codeprints from 11,098 malware binaries, covering 4,123 unique Windows APIs, making it the largest publicly available dataset of this type. RINSER successfully discovered 65 obfuscated Windows APIs related to C2 communication, spying, and evasion in our dataset, which the commercial disassembler IDA failed to identify. Furthermore, we compared RINSER against three state-of-the-art approaches, showing over 20% higher prediction accuracy. We also demonstrated RINSER's resilience to adversarial attacks, including instruction randomization and code displacement, with a performance drop of no more than 3%.

Problem

• Predicting obfuscated API names in malware binaries
• Automating API identification to reduce manual analysis effort
• Handling stripped binaries and adversarial code manipulations

Innovation

• Uses masked language model for API prediction
• Introduces API codeprints from assembly instructions
• Achieves high accuracy on obfuscated binaries