MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces

2025-10-09

AI Summary

To address the challenge of conducting comprehensive user research early in UI design due to resource constraints, this paper presents the first systematic evaluation of multimodal large language models (MLLMs)—specifically GPT-4o, Claude, and Llama—in predicting human subjective interface assessments. We construct a benchmark dataset comprising human perceptual annotations across multiple dimensions (e.g., aesthetic appeal, usability, trustworthiness) for 30 crowdsourced UIs, and quantify alignment between model outputs and human preferences using rank correlation. Results show strong agreement (ρ > 0.7) on surface-level dimensions such as visual attractiveness, yet substantial divergence on deeper UX dimensions like behavioral predictability. This work pioneers the conceptualization of MLLMs as early-stage UX evaluation agents, demonstrating their viability as lightweight design screening tools while rigorously characterizing current limitations in multimodal subjective experience modeling and identifying concrete directions for improvement.

Technology Category

• Humans and AI: User Experience and Usability
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In an ideal design pipeline, user interface (UI) design is intertwined with user research to validate decisions, yet studies are often resource-constrained during early exploration. Recent advances in multimodal large language models (MLLMs) offer a promising opportunity to act as early evaluators, helping designers narrow options before formal testing. Unlike prior work that emphasizes user behavior in narrow domains such as e-commerce with metrics like clicks or conversions, we focus on subjective user evaluations across varied interfaces. We investigate whether MLLMs can mimic human preferences when evaluating individual UIs and comparing them. Using data from a crowdsourcing platform, we benchmark GPT-4o, Claude, and Llama across 30 interfaces and examine alignment with human judgments on multiple UI factors. Our results show that MLLMs approximate human preferences on some dimensions but diverge on others, underscoring both their potential and limitations in supplementing early UX research.

Problem

• Benchmarking MLLMs for predicting human UI perception
• Evaluating MLLMs' ability to mimic human preferences
• Assessing MLLM-human alignment across multiple UI factors

Innovation

• MLLMs predict human UI perception
• Benchmark models on subjective evaluations
• Compare MLLM-human alignment across dimensions