On the Viability of Requirements Generation From Code: An Experience Report

2026-06-24

AI Summary

High-quality paired requirements-code datasets are scarce, significantly hindering empirical research in requirements engineering. This work proposes an agent-based approach that integrates large language models (LLMs), retrieval-augmented generation (RAG), and human-in-the-loop validation to systematically evaluate, for the first time, the feasibility of LLMs in automatically generating realistic requirements from source code, synthesizing requirement smells, and identifying unimplemented requirements. The study reveals that current LLMs struggle to reliably generate or detect such requirement-related issues, and that neither fully automated nor purely manual review alone is sufficient. These findings underscore substantial challenges in automatically constructing trustworthy requirements-code datasets and highlight the critical need for enhanced human oversight in the process.

Abstract

Empirical research in Requirements Engineering is hampered by a lack of adequate datasets that pair source code with corresponding requirements. A tempting route to addressing this lack is the use of Large Language Models to synthesize requirements from existing code bases. We investigate this question by evaluating an LLM-based and RAG-supported agentic approach that generates requirements from source code, verifies their implementation status relying on a human-in-the-loop, and synthetically introduces requirements smells and non-implemented requirements. Our goal was to create datasets that mimic reality and foster empirical RE research. However, during the study, various problems arose, leading to this experience report. Contrary to our initial hypotheses, LLMs were unable to (i) generate non-implemented requirements reliably, (ii) generate high quality requirements, and (iii) reliably introduce synthetic requirements smells. Furthermore, neither an LLM nor a single human-in-the-loop suffices to detect requirements smells reliably. These findings suggest that the generation of code-to-requirements datasets using LLMs is not yet viable and requires human supervision, especially for quality assurance. We critically reflect on our lessons learned and draw relevant conclusions for both researchers and practitioners.

Problem

• Requirements Engineering
• Dataset Generation
• Large Language Models
• Code-to-Requirements
• Empirical Research

Innovation

• Large Language Models
• Requirements Generation
• Code-to-Requirements
• Requirements Smells
• Human-in-the-Loop