Evaluating LLM-Based Goal Extraction in Requirements Engineering: Prompting Strategies and Their Limitations

2026-04-24

AI Summary

This study addresses the limitations of existing approaches in automatically extracting functional goals from software documentation to support Goal-Oriented Requirements Engineering (GORE), particularly concerning accuracy and generalizability. The authors propose a three-stage LLM prompting pipeline—comprising actor identification, high-level goal extraction, and low-level goal extraction—augmented with a dual-LLM generate-and-critique feedback mechanism. They systematically evaluate the impact of zero-shot and few-shot in-context learning strategies. Experimental results demonstrate that the zero-shot feedback approach outperforms standalone few-shot methods, highlighting prompt design as a critical performance bottleneck. The method achieves 61% accuracy in low-level goal extraction, suggesting its primary utility as an assistive tool to accelerate manual extraction rather than fully automate it. Future work will integrate Retrieval-Augmented Generation (RAG) and Chain-of-Thought techniques to further enhance effectiveness.

Technology Category

• Natural Language Processing: Information Extraction
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Due to the textual and repetitive nature of many Requirements Engineering (RE) artefacts, Large Language Models (LLMs) have proven useful to automate their generation and processing. In this paper, we discuss a possible approach for automating the Goal-Oriented Requirements Engineering (GORE) process by extracting functional goals from software documentation through three phases: actor identification, high and low-level goal extraction. To implement these functionalities, we propose a chain of LLMs fed with engineered prompts. We experimented with different variants of in-context learning and measured the similarities between input data and in-context examples to better investigate their impact. Another key element is the generation-critic mechanism, implemented as a feedback loop involving two LLMs. Although the pipeline achieved 61% accuracy in low-level goal identification, the final stage, these results indicate the approach is best suited as a tool to accelerate manual extraction rather than as a full replacement. The feedback-loop mechanism with Zero-shot outperformed stand-alone Few-shot, with an ablation study suggesting that performance slightly degrades without the feedback cycle. However, we reported that the combination of the feedback mechanism with Few-shot does not deliver any advantage, possibly suggesting that the primary performance ceiling is the prompting strategy applied to the 'critic' LLM. Together with the refinement of both the quantity and quality of the Shot examples, future research will integrate Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT) prompting to improve accuracy.

Problem

• Goal-Oriented Requirements Engineering
• Large Language Models
• Prompting Strategies
• Goal Extraction
• Requirements Engineering

Innovation

• Goal-Oriented Requirements Engineering
• Large Language Models
• Prompt Engineering
• Generation-Critic Feedback Loop
• In-Context Learning