Should We Dangle a Carrot? The Effect of Performance-based Incentives in Visualization Experiments

2026-07-08

AI Summary

This study investigates whether monetary incentives influence user task performance and research conclusions in visualization experiments. Through two between-subjects crowdsourced studies, the authors examine the effects of performance-based incentives on both low-level perceptual tasks—specifically, correlation judgments in scatterplots and parallel coordinates—and high-level reasoning tasks involving weather forecasting decisions based on interval or density plots, while manipulating both visual representation and incentive conditions. As the first systematic evaluation of the practical impact of performance incentives in visualization research, this work challenges the common assumption that incentives necessarily enhance performance on higher-order tasks. The results reveal no statistically significant effect of monetary incentives on task performance in either experiment, suggesting that such incentives may be less effective in visualization studies than commonly presumed.

Abstract

A perennial research question in visualization involves identifying which visual encodings for a particular dataset are most effective for users in performing a specific task. The relative effectiveness of the different encodings are commonly identified through controlled experiments. However, designing an experiment involves making many, often ad hoc, decisions about the experimental setup such as whether to include a training module, whether to provide performance-based incentives to participants, etc. Yet, there is limited guidance on how these decisions should be made, and we do not fully understand the impact of these subjective decisions on empirical results. In this paper, we investigate the impact of one such key design decision: monetary rewards. Specifically, we ask: does providing or not providing participants with performance-based financial incentives affect the results and the conclusions that we draw from visualization studies? We conducted two crowdsourced studies investigating the impact of incentives on (i) a low-level, perceptual task (perception of correlations in scatterplots or parallel coordinate plots), and (ii) a task involving reasoning (decision-making based on a weather forecast represented as intervals or density plots). In each of these studies, we manipulate both the visual representation and the presence of incentives as between-subject conditions. We expected to find no effect of incentives on the perceptual task, but to see an effect for the decision-making task. However, we found no effect on task performance in either study. While these are results of only two studies and should be replicated, they suggest that performance-based financial incentives may not always have the intended effect on participants that we presumed, and calls for a reflection of how incentivized studies should be designed.

Problem

• performance-based incentives
• visualization experiments
• experimental design
• crowdsourced studies
• task performance

Innovation

• performance-based incentives
• visualization experiments
• crowdsourced studies
• experimental design
• visual encodings