Technical Report: Full-Stack Fine-Tuning for the Q Programming Language

2025-08-09

AI Summary

General-purpose large language models exhibit weak support for Q—a niche programming language widely used in quantitative finance but severely underrepresented in web-scale training corpora. Method: We introduce the first LeetCode-style benchmark dedicated to Q and propose a full-stack adaptation framework tailored for low-resource programming languages, integrating incremental pretraining, supervised fine-tuning, and reinforcement learning to systematically enhance five parameter-scaled variants of the Qwen-2.5 series. Contribution/Results: The framework demonstrates cross-domain transferability, extending to other low-resource languages or subjective evaluation tasks. Experimental results show that the best-performing model achieves 59% pass@1 accuracy on the Q benchmark—outperforming Claude Opus-4 by +29.5% and GPT-4.1—and all model scales attain significant performance gains.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Even though large language models are becoming increasingly capable, it is still unreasonable to expect them to excel at tasks that are under-represented on the Internet. Leveraging LLMs for specialized applications, particularly in niche programming languages and private domains, remains challenging and largely unsolved. In this work, we address this gap by presenting a comprehensive, open-source approach for adapting LLMs to the Q programming language, a popular tool in quantitative finance that is much less present on the Internet compared to Python, C, Java, and other ``mainstream" languages and is therefore not a strong suit of general-purpose AI models. We introduce a new Leetcode style evaluation dataset for Q, benchmark major frontier models on the dataset, then do pretraining, supervised fine tuning, and reinforcement learning to train a suite of reasoning and non-reasoning models based on the Qwen-2.5 series, spanning five parameter sizes (1.5B, 3B, 7B, 14B, 32B). Our best model achieves a pass@1 accuracy of 59 percent on our Q benchmark, surpassing the best-performing frontier model, Claude Opus-4 by 29.5 percent. Additionally, all models, even our 1.5B model, outperform GPT-4.1 on this task. In addition to releasing models, code, and data, we provide a detailed blueprint for dataset construction, model pretraining, supervised fine-tuning, and reinforcement learning. Our methodology is broadly applicable, and we discuss how these techniques can be extended to other tasks, including those where evaluation may rely on soft or subjective signals.

Problem

• Adapting LLMs to niche Q programming language
• Addressing under-representation of Q in general AI models
• Improving accuracy for specialized quantitative finance tasks

Innovation

• Full-stack fine-tuning for Q programming language
• Leetcode-style evaluation dataset for Q
• Pretraining, SFT, RL for reasoning models