A New Benchmark for Evaluating Code Translation with Third-Party Libraries

2025-09-15

AI Summary

Existing code translation benchmarks inadequately support third-party libraries (TPLs), obscuring LLM failures in library-dependent scenarios. To address this, we introduce TransLibEval—the first TPL-oriented code translation benchmark—comprising 200 real-world tasks across Python, Java, and C++, spanning data processing, machine learning, and web development. We design the first evaluation framework dedicated to TPL-aware translation, integrating high-coverage unit testing with fine-grained dependency analysis. Our evaluation reveals that mainstream LLMs suffer over a 60% average accuracy drop in library-aware translation and uncovers previously overlooked third-party reference errors. Through controlled experiments comparing six translation strategies—including IR-guided and retrieval-augmented approaches—we demonstrate significant performance disparities among them. This work establishes an empirical foundation and actionable optimization pathways for advancing library-aware code intelligence.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In recent years, Large Language Models (LLMs) have been widely studied in the code translation field on the method, class, and even repository levels. However, most of these benchmarks are limited in terms of Third-Party Library (TPL) categories and scales, making TPL-related errors hard to expose and hindering the development of targeted solutions. Considering the high dependence (over 90%) on TPLs in practical programming, demystifying and analyzing LLMs' code translation performance involving various TPLs becomes imperative. To address this gap, we construct TransLibEval, the first benchmark dedicated to library-centric code translation. It consists of 200 real-world tasks across Python, Java, and C++, each explicitly involving TPLs from diverse categories such as data processing, machine learning, and web development, with comprehensive dependency coverage and high-coverage test suites. We evaluate seven recent LLMs of commercial, general, and code-specialized families under six translation strategies of three categories: Direct, IR-guided, and Retrieval-augmented. Experimental results show a dramatic performance drop compared with library-free settings (average CA decline over 60%), while diverse strategies demonstrate heterogeneous advantages. Furthermore, we analyze 4,831 failed cases from GPT-4o, one of the State-of-the-Art (SOTA) LLMs, revealing numerous third-party reference errors that were obscured previously. These findings highlight the unique challenges of library-centric translation and provide practical guidance for improving TPL-aware code intelligence.

Problem

• Evaluating code translation performance with third-party libraries
• Addressing limited benchmark coverage for library dependencies
• Analyzing LLM errors in real-world TPL-centric tasks

Innovation

• Developed TransLibEval benchmark for library translation
• Evaluated LLMs across multiple strategies and languages
• Analyzed TPL reference errors in translation failures