Beyond BLEU: A Semantic Evaluation Method for Code Translation

2026-05-06

AI Summary

Existing code translation evaluation metrics, such as BLEU, rely solely on syntactic similarity and fail to capture semantic correctness. This work introduces, for the first time, the principles of compiler testing into the evaluation of large language models (LLMs) for code translation, proposing a semantic equivalence verification framework grounded in execution consistency. The study defines "semantic accuracy" as the core evaluation metric and develops an LLM-based decompiler that integrates execution trace comparison, semantic equivalence validation, and LLM fine-tuning. Experimental results demonstrate that the proposed LLM decompiler significantly outperforms heuristic baselines, while revealing a negligible correlation between BLEU scores and semantic accuracy (r = –0.127 to 0.354), thereby confirming BLEU’s inadequacy as a proxy for functional correctness in code translation tasks.

Abstract

Code translation is one of the core capabilities of LLMs. However, evaluating the correctness of translations remains difficult, as commonly used metrics such as BLEU measure only syntactic similarity, disregarding program semantics. We propose a novel evaluation methodology for code translation tasks, emphasizing semantic equivalence over surface-level string similarity. Our approach applies established compiler testing methodology to a new domain, allowing the assessment of an LLM fine-tuned for binary lifting tasks (i.e. decompiling binaries to higher-level representations). We introduce a semantic correctness score, defined as the proportion of translations that produce correct execution outcomes, and demonstrate its application by evaluating LLM-based and heuristic decompilers. Our findings show that LLM-based approaches significantly outperform heuristic ones, while BLEU scores show negligible correlation with semantic correctness (r = -0.127 to 0.354), demonstrating that syntactic metrics fail to predict functional accuracy.

Problem

• code translation
• semantic evaluation
• BLEU
• semantic correctness
• program semantics

Innovation

• semantic evaluation
• code translation
• binary lifting
• LLM-based decompilation
• execution correctness