Enhancing Preference-based Linear Bandits via Human Response Time

2024-09-09

arXiv.org

AI Summary

In binary-preference learning, existing methods capture only preference direction but fail to quantify preference strength. To address this information bottleneck, we propose a psychophysiological modeling approach that incorporates human response times (RTs) into preference learning. Specifically, we introduce the EZ diffusion model—the first application of its kind—into the preference-based linear bandits framework, enabling joint modeling of choice outcomes and RTs, as well as joint estimation of the underlying utility function. We theoretically establish that, under strong-preference regimes, RTs provide statistically significant additional information gain. In fixed-budget best-arm identification tasks, our method substantially accelerates convergence. Empirical evaluation across three real-world datasets demonstrates an average 37% reduction in utility estimation error and a marked improvement in best-arm identification accuracy.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

Interactive preference learning systems infer human preferences by presenting queries as pairs of options and collecting binary choices. Although binary choices are simple and widely used, they provide limited information about preference strength. To address this, we leverage human response times, which are inversely related to preference strength, as an additional signal. We propose a computationally efficient method that combines choices and response times to estimate human utility functions, grounded in the EZ diffusion model from psychology. Theoretical and empirical analyses show that for queries with strong preferences, response times complement choices by providing extra information about preference strength, leading to significantly improved utility estimation. We incorporate this estimator into preference-based linear bandits for fixed-budget best-arm identification. Simulations on three real-world datasets demonstrate that using response times significantly accelerates preference learning compared to choice-only approaches. Additional materials, such as code, slides, and talk video, are available at https://shenlirobot.github.io/pages/NeurIPS24.html

Problem

• User Preference Intensity
• Choice Test Accuracy
• Binary Selection

Innovation

• Reaction Time
• Linear Bandit Model
• Preference Prediction