Visual Bias in Simulated Users: The Impact of Luminance and Contrast on Reinforcement Learning-based Interaction

2026-03-02

AI Summary

This study investigates how visual rendering artifacts—specifically luminance and contrast—affect reinforcement learning–driven simulated users in human-computer interaction, leading to behavioral biases. The authors systematically manipulate the luminance and contrast of task objects, distractors, and backgrounds, training 247 simulated users under no-interference, static-interference, and dynamic-interference conditions to evaluate performance and generalization in pointing and tracking tasks. The work reveals, for the first time, that the relative ordering of luminance—not absolute values—is critical for user robustness: static interference substantially degrades performance, whereas motion cues mitigate this effect. Although extreme luminance levels (e.g., pure black) enhance task-specific performance, they impair generalization. These findings demonstrate that minor luminance variations can significantly alter learned behaviors, and maintaining consistent luminance relationships improves model robustness.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Humans and AI: Intelligent User Interfaces
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Reinforcement learning (RL) enables simulations of HCI tasks, yet their validity is questionable when performance is driven by visual rendering artifacts distinct from interaction design. We provide the first systematic analysis of how luminance and contrast affect behavior by training 247 \RV{simulated users using RL} on pointing and tracking tasks. We vary the luminance of task-relevant objects, distractors, and background under no distractor, static distractor, and moving distractor conditions, and evaluate task performance and robustness to unseen luminances. Results show luminance becomes critical with static distractors, substantially degrading performance and robustness, whereas motion cues mitigate this issue. Furthermore, robustness depends on preserving relational ordering between luminances rather than matching absolute values. Extreme luminances, especially black, often yield high performance but poor robustness. Overall, seemingly minor luminance changes can strongly shape learned behavior, revealing critical insights into what RL-driven simulated users actually learn.

Problem

• visual bias
• luminance
• contrast
• reinforcement learning
• simulated users

Innovation

• visual bias
• reinforcement learning
• luminance
• contrast
• simulated users