Recommending Usability Improvements with Multimodal Large Language Models

2026-04-28

AI Summary

This study addresses the high cost and expert dependency of traditional usability evaluations, which often hinder adoption by small development teams. It presents the first application of multimodal large language models (MLLMs) to automated usability analysis, leveraging screenshots and user interaction recordings to automatically detect usability issues based on Nielsen’s heuristics. The approach generates actionable improvement suggestions and incorporates severity-based prioritization to alleviate developers’ burden in determining issue importance. Findings from a user study indicate that software engineers perceive the high-priority recommendations as both high-quality and highly practical, demonstrating the method’s effectiveness as a low-cost, accessible complement to conventional usability assessment practices.

Abstract

Usability describes quality attributes of application user interfaces that determine how effectively users can interact with them. Traditional usability evaluation methods require considerable expertise and resources, which can be challenging, especially for small teams and organizations. Automating usability evaluation could make it more accessible and help to improve the user experience. The recent emergence of powerful multimodal large language models (MLLMs) has opened new opportunities for automating usability evaluation and recommendation of improvements. These models can process visual inputs such as images and videos alongside textual context, which enables the identification of usability issues and the generation of actionable suggestions to resolve these issues. In this paper, we present a novel automated approach that uses limited application context and screen recordings of user interactions as input to an MLLM. The model automatically identifies and describes usability issues based on Nielsens usability heuristics, and provides corresponding explanations and improvement recommendations. To reduce the developer effort of manual prioritization, the recommendations are ranked by severity. The quality and practical usefulness of the generated recommendations were evaluated based on a user study that involved software engineers as participants. The evaluation focused on the highest-ranked suggestions provided by the model. The results demonstrate the potential of our approach to provide low-effort usability improvement recommendations. This makes it a promising complement to traditional evaluation methods, especially in settings with limited access to usability experts. In this sense, the approach serves as a basis for future integration into development tools to enable automated usability evaluation within software engineering workflows.

Problem

• usability evaluation
• multimodal large language models
• user interface
• automated recommendation
• software engineering

Innovation

• Multimodal Large Language Models
• Automated Usability Evaluation
• Usability Heuristics
• Screen Recording Analysis
• Severity-based Recommendation Ranking