GUISpector: An MLLM Agent Framework for Automated Verification of Natural Language Requirements in GUI Prototypes

2025-10-06

AI Summary

Verifying consistency between GUI prototypes and natural-language (NL) requirements remains challenging due to manual effort, delayed feedback, and poor integration with automated development pipelines. Method: This paper introduces the first MLLM-based agent framework for GUI requirement verification, integrating NL understanding, GUI state parsing, reasoning trajectory planning, and visual feedback generation to enable end-to-end autonomous testing and actionable feedback—fully embedded within an LLM-driven development pipeline. Contribution/Results: Evaluated on a benchmark of 900 acceptance criteria covering 150 functional requirements, the framework achieves high accuracy in detecting requirement satisfaction or violation. It demonstrates strong robustness across diverse GUI applications and significant engineering integration potential, markedly improving automation, traceability, and efficiency in requirement-to-implementation consistency verification.

Technology Category

• Natural Language Processing: Safety and Robustness
• Humans and AI: User Experience and Usability
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

GUIs are foundational to interactive systems and play a pivotal role in early requirements elicitation through prototyping. Ensuring that GUI implementations fulfill NL requirements is essential for robust software engineering, especially as LLM-driven programming agents become increasingly integrated into development workflows. Existing GUI testing approaches, whether traditional or LLM-driven, often fall short in handling the complexity of modern interfaces, and typically lack actionable feedback and effective integration with automated development agents. In this paper, we introduce GUISpector, a novel framework that leverages a multi-modal (M)LLM-based agent for the automated verification of NL requirements in GUI prototypes. First, GUISpector adapts a MLLM agent to interpret and operationalize NL requirements, enabling to autonomously plan and execute verification trajectories across GUI applications. Second, GUISpector systematically extracts detailed NL feedback from the agent's verification process, providing developers with actionable insights that can be used to iteratively refine the GUI artifact or directly inform LLM-based code generation in a closed feedback loop. Third, we present an integrated tool that unifies these capabilities, offering practitioners an accessible interface for supervising verification runs, inspecting agent rationales and managing the end-to-end requirements verification process. We evaluated GUISpector on a comprehensive set of 150 requirements based on 900 acceptance criteria annotations across diverse GUI applications, demonstrating effective detection of requirement satisfaction and violations and highlighting its potential for seamless integration of actionable feedback into automated LLM-driven development workflows. The video presentation of GUISpector is available at: https://youtu.be/JByYF6BNQeE, showcasing its main capabilities.

Problem

• Automating verification of natural language requirements in GUI prototypes
• Addressing limitations of existing GUI testing approaches for modern interfaces
• Providing actionable feedback for GUI refinement and LLM-based code generation

Innovation

• Uses MLLM agent for GUI requirements verification
• Extracts actionable feedback from verification process
• Integrates tool for end-to-end verification management