Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We?

2026-05-24

AI Summary

Existing log generation methods are predominantly evaluated in monolingual settings, leaving their cross-lingual effectiveness unclear. This work constructs a multilingual logging benchmark comprising 150,000 instances across five programming languages and presents the first systematic evaluation of three state-of-the-art approaches and five large language models in cross-lingual log generation. The study reveals significant disparities in logging difficulty across languages, attributed to language-specific logging insertion patterns and idioms, underscoring the need for tailored multilingual logging strategies. Experimental results demonstrate that UniLog achieves the strongest overall performance, with JavaScript proving more amenable to log generation than Python, which poses greater challenges. Moreover, merely scaling model size or training data yields limited gains in multilingual logging efficacy.

Abstract

Log statements capture critical information for software maintenance activities such as testing, debugging, and failure analysis. Because of this importance, developers must carefully design log statements, which requires significant effort. To support developers, various end-to-end automated log statement generation approaches have been proposed, whereas these approaches have mainly been evaluated within a single programming language environment and their effectiveness in multilingual environments remains underexplored. In this paper, we therefore comparatively evaluate three state-of-the-art log statement generation approaches and five large language models (LLMs) across multiple programming languages. For this purpose, we constructed a multilingual benchmark comprising 150,000 instances across five programming languages. Our empirical results demonstrate that UniLog, a state-of-the-art approach, achieves the best overall performance, maintaining high effectiveness even in multilingual environments. We also observe substantial variance in the difficulty of log generation across languages: Python presents a greater challenge, whereas JavaScript yields comparatively better performance. Detailed analysis reveals that these disparities stem from variations in log insertion distributions and language-specific logging idioms. Our findings indicate that simply scaling model size or the volume of training data is insufficient for multilingual log generation; rather, designing approaches tailored to the specific characteristics of target languages is crucial. These findings suggest that future automated logging techniques should explicitly account for language-specific logging characteristics to achieve robust performance in multilingual software development environments.

Problem

• log statement generation
• multilingual scenarios
• software maintenance
• programming languages
• language-specific logging

Innovation

• log statement generation
• multilingual benchmark
• large language models
• language-specific logging idioms
• automated software logging