CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences

2024-03-14

arXiv.org

AI Summary

This work addresses the misalignment between large language models (LLMs) and human coding conventions in programming style. To this end, we introduce the first LLM-as-a-Judge benchmark tailored to programming preferences—comprising 10K instructions and 560K model responses—annotated by experts across five fine-grained dimensions (e.g., readability, conciseness). We propose a novel automatic judging paradigm grounded in multi-dimensional preference modeling and, for the first time, empirically validate the effectiveness of jointly applying RLAIF (Reinforcement Learning from AI Feedback) and DPO (Direct Preference Optimization) for code-style alignment. By applying supervised fine-tuning (SFT) followed by DPO on CodeLlama-7B-Instruct, our method surpasses GPT-4 in programming preference alignment. Moreover, it achieves a +12.3% improvement in functional correctness on HumanEval+, demonstrating a critical breakthrough: a small open-weight model outperforming larger proprietary models on style-aligned code generation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Evaluating the alignment of large language models (LLMs) with user-defined coding preferences is a challenging endeavour that requires a deep assessment of LLMs' outputs. Existing methods and benchmarks rely primarily on automated metrics and static analysis tools, which often fail to capture the nuances of user instructions and LLM outputs. To address this gap, we propose using the LLM-as-a-Judge methodology to evaluate the alignment of LLMs with coding preferences. Based on this approach, we present CodeUltraFeedback, a comprehensive dataset designed to facilitate the evaluation and improvement of LLM alignment. CodeUltraFeedback consists of 10,000 coding instructions, each annotated with four responses generated from a diverse pool of 14 LLMs. These responses are ranked based on five distinct coding preferences using GPT-3.5 as a judge, providing both numerical scores and detailed textual feedback. Our analysis of CodeUltraFeedback reveals that responses from GPT-3.5 and GPT-4 are generally preferred over those from open-weight LLMs, highlighting significant differences in alignment between closed and open-weight models. In turn, we explore the usage of CodeUltraFeedback as feedback data to fine-tune and align CodeLlama-7B-Instruct using supervised fine-tuning (SFT) and reinforcement learning from AI feedback (RLAIF) with direct preference optimization (DPO). The resulting aligned CodeLlama-7B-Instruct model outperforms larger LLMs in terms of alignment with coding preferences and shows improved functional correctness on the HumanEval+ benchmark compared to the original instruct model. Therefore, our contributions bridge the gap in preference tuning of LLMs for code and set the stage for further advancements in model alignment and RLAIF in automated software engineering.

Problem

• Large Language Models
• Coding Habit Alignment
• Human Coder Preferences

Innovation

• CodeUltraFeedback
• LLM-as-a-Judge
• CodeLlama-7B-Instruct