🤖 AI Summary
This study addresses the challenge users face in accurately evaluating whether large language models genuinely reduce time spent on cognitive tasks during human-AI collaboration. Through a pre-registered, large-scale behavioral experiment (N = 1,237), the authors compared task completion time and perceived mental effort across three conditions: working independently, with AI assistance, and with human assistance. The results reveal, for the first time, an “acceleration illusion”: participants consistently overestimated the time savings afforded by AI, despite no significant difference in actual completion time relative to independent work. Notably, subjective effort was significantly lower under AI assistance, indicating that perceived efficiency cannot be captured by objective time metrics alone. These findings challenge prevailing paradigms that prioritize temporal measures in assessing AI effectiveness.
📝 Abstract
Large language models (LLMs) have the potential to boost human productivity by speeding up task completion -- provided users know when to offload cognitive work to them. But we do not know if users are well-calibrated in estimating these potential time savings. We conducted a preregistered large-scale behavioral study (N = 1237) to characterize mismatches between expectations and reality, with a focus on simple cognitive tasks. While actual completion times between independent completion and AI-assisted completion did not differ, participants predicted AI to be significantly faster. The same bias was not observed when imagining help from another human participant. We identify a speedup illusion where people have accurate forecasts of independent completion times but significantly underestimate AI-assisted times. Additionally, time and effort dissociate: participants reported lower subjective effort with AI despite equivalent completion times. This suggests that completion time itself is not sufficient to characterize efficiency gains.