CODEFUSE-DEBENCH: An Empirical Study on Readability, Recompilability, and Functionality

2026-05-28

AI Summary

Existing decompilation evaluations predominantly rely on syntactic similarity or isolated readability metrics, which inadequately capture the practical reusability of recovered code. To address this limitation, this work proposes a three-dimensional evaluation paradigm centered on reusability—encompassing readability, recompilability, and functionality—and introduces DEBENCH, the first automated multidimensional benchmark comprising 240 atomic functions and 640 binary samples. Leveraging LLM-as-judge for readability scoring, URAF fine-grained metrics, 50-round iterative compilation repair, and Frida-driven multilevel dynamic differential tracing, the study systematically uncovers significant discrepancies across evaluation dimensions: only 1.2% of outputs from the best decompiler–LLM combination achieve full functional equivalence; Clang-generated code exhibits 2.6× higher functionality than GCC’s; and functional recovery capability varies by up to 20× across decompilers. The analysis further identifies three dominant failure modes, including type system collapse.

Abstract

Binary decompilation aims to recover binaries into high-level source code, but existing evaluations mainly rely on syntactic similarity or single-axis readability metrics, which fail to capture practical reusability. We propose a reusability-driven evaluation paradigm that measures decompiler quality along three orthogonal dimensions: readability, recompilability, and functionality. We present DEBENCH, the first automated framework for multidimensional decompilation evaluation. DEBENCH contains 240 atomic test functions, organized into 8 source files and compiled into 640 binaries. It combines LLM-as-judge readability scoring with URAF (18 sub-dimensions), iterative compile-and-repair under a fixed 50-iteration budget, and Frida-based differential dynamic tracing at the program, function, and instruction levels. We evaluate five mainstream decompilers and three repair LLMs. Our study reveals four findings. First, the reusability cliff is steep: the best decompiler-LLM pair reaches 22.3% Exact+Partial program-level behavioral overlap but only 1.2% exact stdout match, nearly 50 points below recompilability. Second, settings that maximize readability do not maximize functionality: -O3 yields the lowest readability but the highest functionality, and Clang gives lower readability than GCC but 2.6x higher functionality. Third, cross-decompiler variation at the functional level is 20x, far larger than the 1.6x cross-LLM variation, showing that progress depends more on decompiler engines than larger repair models. Fourth, failures fall into three categories: syntactic noise, type-system collapse (about 19% of repair errors), and irreversible upstream losses such as ARM64 relocation idioms and C++ ABI features.

Problem

• binary decompilation
• reusability
• evaluation
• readability
• functionality

Innovation

• decompilation evaluation
• reusability-driven paradigm
• multidimensional benchmarking
• dynamic differential tracing
• LLM-assisted repair