🤖 AI Summary
Despite the growing adoption of pre-trained code language models (Code-PLMs), their capabilities on the R programming language—particularly under its dual-paradigm ecosystem (Tidyverse vs. Base R)—remain systematically unexplored. Method: We introduce the first open-source, style-annotated R code dataset and conduct a comprehensive empirical study across four axes: cross-paradigm evaluation, cross-project generalization, multilingual fine-tuning ablation, and human evaluation—focusing on code summarization and method name prediction. Contribution/Results: (1) Syntax paradigm significantly impacts model performance, especially degrading summarization quality; (2) R-specific contextual dependencies severely hinder cross-project transfer; (3) Generic multilingual fine-tuning does not consistently improve R-specific task performance; (4) All evaluated Code-PLMs exhibit performance degradation on R tasks, corroborated by human assessment indicating low-quality generated summaries. This work establishes the first benchmark, dataset, and empirical foundation for R-oriented code intelligence research.
📝 Abstract
Recent advancements in developing Pre-trained Language Models for Code (Code-PLMs) have urged many areas of Software Engineering (SE) and brought breakthrough results for many SE tasks. Though these models have achieved the state-of-the-art performance for SE tasks for many popular programming languages, such as Java and Python, the Scientific Software and its related languages like R programming language have rarely benefited or even been evaluated with the Code-PLMs. Research has shown that R has many differences with other programming languages and requires specific techniques. In this study, we provide the first insights for code intelligence for R. For this purpose, we collect and open source an R dataset, and evaluate Code-PLMs for the two tasks of code summarization and method name prediction using several settings and strategies, including the differences in two R styles, Tidy-verse and Base R. Our results demonstrate that the studied models have experienced varying degrees of performance degradation when processing R programming language code, which is supported by human evaluation. Additionally, not all models show performance improvement in R-specific tasks even after multi-language fine-tuning. The dual syntax paradigms in R significantly impact the models' performance, particularly in code summarization tasks. Furthermore, the project-specific context inherent in R codebases significantly impacts the performance when attempting cross-project training.