REBENCH: A Procedural, Fair-by-Construction Benchmark for LLMs on Stripped-Binary Types and Names (Extended Version)

2026-04-29

AI Summary

Current large language models lack a standardized evaluation benchmark for binary reverse engineering tasks—such as function and variable name recovery and type inference—making it difficult to fairly assess their capabilities. To address this gap, this work proposes REBench, the first fair and reproducible reverse engineering benchmark. REBench leverages knowledge-base-driven data construction and programmatic generation methods to integrate hundreds of millions of lines of source code with diverse binaries spanning multiple architectures and optimization levels. It employs byte-level stack trace analysis to produce ground-truth labels that preserve task complexity while enabling cross-platform, generalizable evaluation. Empirical results demonstrate that state-of-the-art large language models still face significant challenges on these intricate reverse engineering tasks.

Abstract

Large Language Models (LLMs) have achieved remarkable progress in recent years, driving their adoption across a wide range of domains, including computer security. In reverse engineering, LLMs are increasingly applied to critical tasks such as function and variable name recovery and type inference. However, despite the rapid growth of research in this area, progress has been hindered by the absence of a standardized dataset. Existing studies rely on disparate datasets, preprocessing pipelines, and evaluation metrics, making fair comparisons between approaches difficult and obscuring a clear understanding of LLM capabilities in binary analysis. To address these challenges, we present REBench, a comprehensive benchmark dataset for evaluating LLMs on binary reverse engineering tasks. REBench consolidates a superset of existing datasets, comprising hundreds of millions of lines of source code and a diverse collection of binaries spanning multiple architectures and optimization levels. REBench adopts a knowledge-base-driven methodology that stores byte-level stack information to generate ground truth, ensuring that task difficulty is preserved while maintaining universal applicability. This design enables fair evaluation across tasks while avoiding simplifications that could bias results. As a use case, we apply REBench to measure the reverse engineering performance of LLMs and the result demonstrates difficulties in complex tasks.

Problem

• Large Language Models
• Binary Reverse Engineering
• Benchmark Dataset
• Function Name Recovery
• Type Inference

Innovation

• REBench
• binary reverse engineering
• fair-by-construction benchmark
• knowledge-base-driven
• LLM evaluation