When Prompt Under-Specification Improves Code Correctness: An Exploratory Study of Prompt Wording and Structure Effects on LLM-Based Code Generation

2026-04-27

AI Summary

Existing research on code generation often relies on prompt benchmarks with simplistic structures, which fail to capture how variations in phrasing and structure affect the robustness of large language models in real-world scenarios. This study systematically investigates how prompt structure, task complexity, and descriptive richness jointly influence model robustness to prompt perturbations, evaluating ten models on HumanEval and LiveCodeBench. The findings reveal that richly structured task descriptions can significantly mitigate—and sometimes even reverse—the negative impact of underspecified prompts. Moreover, specific modifications, such as removing misleading terminology or constraints, actively enhance code correctness, challenging the common assumption that more explicit prompts always yield better performance. On LiveCodeBench, most underspecified perturbations have negligible effects, with some even improving performance through three identified gain mechanisms: breaking overfit terminology, eliminating misleading constraints, and removing spurious trigger tokens.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models are increasingly used for code generation, yet the correctness of their outputs depends not only on model capability but also on how tasks are specified. Prior studies demonstrate that small changes in natural language prompts, particularly under-specification can substantially reduce code correctness; however, these findings are largely based on minimal-specification benchmarks such as HumanEval and MBPP, where limited structural redundancy may exaggerate sensitivity. In this exploratory study, we investigate how prompt structure, task complexity, and specification richness interact with LLM robustness to prompt mutations. We evaluate 10 different models across HumanEval and the structurally richer LiveCodeBench. Our results reveal that robustness is not a fixed property of LLMs but is highly dependent on prompt structure: the same under-specification mutations that degrade performance on HumanEval have near-zero net effect on LiveCodeBench due to redundancy across descriptions, constraints, examples, and I/O conventions. Surprisingly, we also find that prompt mutations can improve correctness. In LiveCodeBench, under-specification often breaks misleading lexical or structural cues that trigger incorrect retrieval-based solution strategies, leading to correctness improvements that counterbalance degradations. Manual analysis identifies consistent mechanisms behind these improvements, including the disruption of over-fitted terminology, removal of misleading constraints, and elimination of spurious identifier triggers. Overall, our study shows that structurally rich task descriptions can substantially mitigate the negative effects of under-specification and, in some cases, even enhance correctness. We outline categories of prompt modifications that positively influence the behavior of LLM code-generation, offering practical insights for writing robust prompts.

Problem

• prompt under-specification
• code correctness
• LLM robustness
• prompt structure
• code generation

Innovation

• prompt under-specification
• code generation
• LLM robustness
• prompt structure
• LiveCodeBench