Prompt Variability Effects On LLM Code Generation

2025-06-11

AI Summary

This work investigates the sensitivity of large language models (LLMs) to prompt variations in code generation tasks and how this sensitivity is modulated by user background—such as software development experience. To address this, we propose the first synthetic evaluation pipeline and a persona-based systematic assessment framework specifically designed for code generation; the framework is task- and model-agnostic. Our methodology comprises synthetic prompt construction, persona modeling grounded in realistic developer profiles, multi-dimensional functional and quality evaluation (e.g., correctness, readability, efficiency), and a cross-model reproducible evaluation protocol. Experimental results demonstrate that prompts conditioned on distinct user personas significantly affect both functional correctness and holistic code quality. Moreover, our framework consistently uncovers latent behavioral biases across diverse LLMs. All evaluation code and artifacts are publicly released to support reproducibility and community advancement.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Code generation is one of the most active areas of application of Large Language Models (LLMs). While LLMs lower barriers to writing code and accelerate development process, the overall quality of generated programs depends on the quality of given prompts. Specifically, functionality and quality of generated code can be sensitive to user's background and familiarity with software development. It is therefore important to quantify LLM's sensitivity to variations in the input. To this end we propose a synthetic evaluation pipeline for code generation with LLMs, as well as a systematic persona-based evaluation approach to expose qualitative differences of LLM responses dependent on prospective user background. Both proposed methods are completely independent from specific programming tasks and LLMs, and thus are widely applicable. We provide experimental evidence illustrating utility of our methods and share our code for the benefit of the community.

Problem

• Quantify LLM sensitivity to prompt variations in code generation
• Evaluate code quality differences based on user background
• Propose task-agnostic methods for assessing LLM code generation

Innovation

• Synthetic evaluation pipeline for LLM code generation
• Persona-based evaluation for user background impact
• Task and LLM independent widely applicable methods