ClarifyCodeBench: Evaluating LLMs on Clarifying Ambiguous Requirements for Code Generation

2026-07-01

AI Summary

This work addresses the limited ability of large language models (LLMs) to handle ambiguous requirements in code generation, a challenge exacerbated by the lack of systematic evaluation of their clarification capabilities. We introduce ClarifyCodeBench, the first interactive benchmark grounded in real-world programming tasks, which features human-annotated ambiguity types, clarification questions, and reference answers to enable structured assessment. To quantify clarification effectiveness, we propose two novel metrics: Turn-discounted Key Question Rate and Optimal Round Adherence. Our comprehensive evaluation reveals that strong code generation performance does not necessarily entail effective clarification; reasoning augmentation offers minimal gains in ambiguity detection; and model performance degrades substantially in multi-ambiguity scenarios—evidence of a decoupling between code generation and clarification abilities.

Abstract

Large Language Models have emerged as programming assistants. However, the efficacy of code generation is constrained by the quality of input requirements, which are frequently ambiguous, incomplete, or underspecified. While LLMs excel at one-shot code synthesis, their ability to proactively clarify intent remains underexplored, as a critical trait for robust software engineering. Existing benchmarks largely overlook this interactive bottleneck, assuming perfectly specified prompts that do not reflect the iterative nature of requirement elicitation. To bridge this gap, we introduce ClarifyCodeBench, a novel interactive benchmark for evaluating LLMs' capability in resolving requirement ambiguity. Constructed from real-world programming tasks, ClarifyCodeBench features high-quality manual annotations, including N unique ambiguity types, associated clarification questions, and corresponding ground-truth answers. Furthermore, we formalize two rigorous metrics to assess the interaction quality: Turn-discounted Key Question Rate, which penalizes inefficient questioning, and Optimal Round Adherence, which measures the precision of the elicitation process. We conduct a systematic evaluation of six state-of-the-art LLMs using ClarifyCodeBench. Our empirical results yield three critical insights: 1) Capability Decoupling: Strong code generation performance does not inherently translate to effective requirement clarification; 2) The Reasoning Paradox: While increased computational thinking enhances code correctness, it yields marginal gains in identifying ambiguities; 3) The Multi-ambiguity Ceiling: LLMs' clarification performance degrades sharply as the density of ambiguities increases, revealing a significant bottleneck in handling complex, real-world specifications. Our work underscores the necessity for future AI4SE research to transition from static synthesis to interactive elicitation.

Problem

• ambiguous requirements
• code generation
• interactive clarification
• requirement elicitation
• LLM evaluation

Innovation

• ClarifyCodeBench
• requirement clarification
• interactive benchmark
• ambiguity resolution
• AI4SE