The Behavioral Fabric of LLM-Powered GUI Agents: Human Values and Interaction Outcomes

2026-01-22

AI Summary

This study investigates how user preferences and human values shape the decision-making and behavior of large language model (LLM)-driven graphical user interface agents. To this end, we develop a standardized benchmark comprising 14 real-world web tasks and systematically inject 12 distinct human values into four state-of-the-art LLM agents, analyzing their interaction trajectories in domains such as shopping and travel. Our empirical findings reveal, for the first time, that value alignment significantly alters agent behavior—prompting more frequent use of relevant filtering functions—yet salient interface cues (e.g., discount notifications) often override these value-driven tendencies, leading to simplified actions accompanied by post-hoc rationalizations. We open-source the benchmark and methodological framework to support future research on value-aligned interactive agents.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Learning Preferences or Rankings
• Philosophy and Ethics of AI: Morality & Value-based AI

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large Language Model (LLM)-powered web GUI agents are increasingly automating everyday online tasks. Despite their popularity, little is known about how users'preferences and values impact agents'reasoning and behavior. In this work, we investigate how both explicit and implicit user preferences, as well as the underlying user values, influence agent decision-making and action trajectories. We built a controlled testbed of 14 common interactive web tasks, spanning shopping, travel, dining, and housing, each replicated from real websites and integrated with a low-fidelity LLM-based recommender system. We injected 12 human preferences and values as personas into four state-of-the-art agents and systematically analyzed their task behaviors. Our results show that preference and value-infused prompts consistently guided agents toward outcomes that reflected these preferences and values. While the absence of user preference or value guidance led agents to exhibit a strong efficiency bias and employ shortest-path strategies, their presence steered agents'behavior trajectories through the greater use of corresponding filters and interactive web features. Despite their influence, dominant interface cues, such as discounts and advertisements, frequently overrode these effects, shortening the agents'action trajectories and inducing rationalizations that masked rather than reflected value-consistent reasoning. The contributions of this paper are twofold: (1) an open-source testbed for studying the influence of values in agent behaviors, and (2) an empirical investigation of how user preferences and values shape web agent behaviors.

Problem

• LLM-powered GUI agents
• user preferences
• human values
• agent behavior
• interaction outcomes

Innovation

• LLM-powered GUI agents
• human values
• preference-guided reasoning
• interactive web tasks
• value-aligned behavior