Assessing the Impact of Requirement Ambiguity on LLM-based Function-Level Code Generation

2026-04-23

AI Summary

This work addresses the critical challenge posed by ambiguity in real-world software requirements, which severely undermines the reliability of large language models (LLMs) in function-level code generation. To enable systematic evaluation, we introduce Orchid—the first benchmark specifically designed for code generation under ambiguous requirements—comprising 1,304 tasks categorized along lexical, syntactic, semantic, and vagueness-related dimensions of ambiguity. Empirical evaluation reveals that all assessed LLMs suffer significant performance degradation under ambiguous specifications, with high-end models exhibiting heightened sensitivity and frequently producing functionally inconsistent code for identical requirements. Crucially, LLMs consistently fail to autonomously detect or resolve such ambiguities. Our study provides the first open-source benchmark and a comprehensive analytical framework to advance research on code generation in the presence of requirement ambiguity.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Software requirement ambiguity is ubiquitous in real-world development, stemming from the inherent imprecision of natural language and the varying interpretations of stakeholders. While Large Language Models (LLMs) have demonstrated impressive capabilities in generating code from precise specifications, such ambiguity poses a significant obstacle to reliable automated code generation. Existing benchmarks typically assume clear and unambiguous requirements, leaving an empirical gap in understanding how LLMs behave when faced with the inherent uncertainty of real-world software requirements. In this paper, we introduce Orchid, the first code generation benchmark specifically designed with ambiguous requirements. It comprises 1,304 function-level tasks covering four distinct types of ambiguity: lexical, syntactic, semantic, and vagueness. Leveraging this dataset, we conduct the first systematic empirical study to evaluate the impact of requirement ambiguity on LLM-based code generation. Our results demonstrate that ambiguity consistently degrades the performance of all evaluated LLMs, with the most pronounced negative effects observed in highly advanced models. Furthermore, we observe that LLMs frequently produce functionally divergent implementations for the same ambiguous requirement and lack the capability to identify or resolve such ambiguity autonomously. These findings reveal a significant performance gap between clear and ambiguous requirements, underscoring the urgent need for ambiguity-aware techniques in the next generation of automated software engineering tools. The Orchid benchmark is publicly available at https://huggingface.co/datasets/SII-YDD/Orchid.

Problem

• requirement ambiguity
• LLM-based code generation
• function-level code generation
• software requirements
• code generation benchmark

Innovation

• requirement ambiguity
• LLM-based code generation
• Orchid benchmark
• function-level tasks
• ambiguity-aware software engineering