Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generation

2026-06-27

AI Summary

This study investigates how large language models can simultaneously satisfy functional requirements (e.g., correctness, executability) and non-functional attributes (e.g., readability, maintainability) in code generation through reward-free alignment methods. It presents the first systematic comparison of alignment efficacy when starting from either pretrained or instruction-tuned models. Leveraging preference data constructed via SelfCodeAlign and employing two reward-free optimization techniques—Direct Preference Optimization (DPO) and Best-of-N with Bootstrapped Objectives (BoNBoN)—the authors conduct empirical evaluations across five state-of-the-art models. Results reveal that alignment from pretrained models yields greater relative improvements, yet their absolute performance remains below that of instruction-tuned baselines. Conversely, aligning instruction-tuned models offers limited gains and occasionally leads to performance degradation, highlighting a trade-off between performance enhancement and stability depending on the model initialization strategy.

Abstract

Large Language Model (LLM) alignment trains an LLM using preference data to produce outputs that better meet established quality standards. While LLM alignment techniques are studied for non-coding tasks, we know little about their usefulness for coding tasks. It is unclear whether LLM code alignment could support both functional requirements (producing executable, correct code) and non-functional requirements (code readability, style, maintainability). It is also unknown whether alignment for a code LLM should begin with base pretrained version or the finetuned (i.e., instruction-tuned) version of the LLM. In this paper, we offer insights on the above two research questions by conducting an empirical study. We studied five state-of-the-art (SOTA) LLMs using two widely used LLM alignment techniques: Direct Preference Optimization (DPO) and BoNBoN. For each training record, we created a preference pair as accepted and rejected instances by using the SelfCodeAlign pipeline. DPO and BoNBoN are reward-free models, i.e., they eliminate the need for multiple reward scores for output preferences. We tuned each LLM using the two alignment techniques in two settings: pretrained and finetuned versions of an LLM. We evaluated functional requirements using four SOTA benchmarks (HumanEval+, MBPP+, EvalPerf, EvoEval) and non-functional requirements using the CODAL benchmark, which evaluates code quality across five dimensions derived from software engineering practices. We find that pretrained-to-aligned pathways achieve larger improvements in the aligned variant over its pretrained variant. But the pretrained variant is generally less accurate than its finetuned variant. However, finetuned- to-aligned offers smaller performance improvements or, in some cases, degradation in the aligned variant than its finetuned variant.

Problem

• LLM alignment
• code generation
• functional requirements
• non-functional requirements
• pretrained vs fine-tuned

Innovation

• reward-free alignment
• code generation
• pretrained vs finetuned
• functional and non-functional requirements
• Direct Preference Optimization