Teaching LLMs a Low-Resource Language: Enhancing Code Completion in Pharo

2026-07-06

AI Summary

This work addresses the longstanding scarcity of effective code completion tools for Pharo, a low-resource programming language hindered by limited training data. We introduce the first Pharo-specific code completion benchmark and propose an end-to-end pipeline that tailors open-source large code models through domain-specific data collection and cleaning, continual pretraining, and instruction fine-tuning. Experimental results demonstrate that our compact, specialized model substantially outperforms the original baseline and even surpasses larger general-purpose code models on real-world GitHub code completion tasks. Furthermore, the model supports real-time integration into integrated development environments (IDEs), thereby validating the efficacy and superiority of domain specialization for low-resource programming languages.

Abstract

Large Language Models (LLMs) unlocked new possibilities in automated code writing, becoming the backbone of most code completion tools. While LLMs excel in mainstream languages, they often lack support for the so-called low-resource languages where training data is scarce. As a result, these languages lag behind in the quality of code completion tooling available to their communities. A concrete example is Pharo, a Smalltalk-inspired language whose IDE currently offers only single-token completion. In this work, we report on our experience bringing LLM-based code completion to Pharo. First, we describe an end-to-end pipeline that combines Pharo-specific data curation, continued pre-training and fine-tuning of open code LLMs. Second, we introduce a set of Pharo code completion benchmarks designed to evaluate whether models (i) learn Pharo's syntax and (ii) accurately complete masked Pharo code from real-world GitHub repositories. Third, we show empirically that Pharo-specialized models substantially outperform their original base checkpoints and also exceed the accuracy of substantially larger code LLMs on Pharo completion. Overall, our case study demonstrates the feasibility of bringing strong LLM-based code completion to low-resource programming languages, with models small enough to provide ``real-time'' in-IDE support.

Problem

• low-resource language
• code completion
• Large Language Models
• Pharo
• programming language

Innovation

• low-resource programming languages
• LLM fine-tuning
• code completion
• Pharo
• domain-specific benchmarks