Improving Human Performance with Value-Aware Interventions: A Case Study in Chess

2026-04-15

AI Summary

This study addresses the challenge of intelligently selecting when and how to intervene with AI assistance in human sequential decision-making to maximize overall performance. The authors propose a value-aware intervention strategy that, for the first time, leverages the inconsistency between policy and value functions as an intervention signal, optimizing human decisions under a limited intervention budget. Grounded in Markov decision processes, the approach integrates a human behavioral model with a reinforcement learning value function and is trained on large-scale chess gameplay data. Experimental results demonstrate that the method significantly outperforms a Stockfish-based baseline in simulated environments. Furthermore, in a human-AI interaction study involving 600 games with 20 players, it effectively enhances performance among low- to mid-skill players while maintaining comparable efficacy for high-skill players.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Intelligent Robots: Human-Robot Interaction
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

AI systems are increasingly used to assist humans in sequential decision-making tasks, yet determining when and how an AI assistant should intervene remains a fundamental challenge. A potential baseline is to recommend the optimal action according to a strong model. However, such actions assume optimal follow-up actions, which human decision makers may fail to execute, potentially reducing overall performance. In this work, we propose and study value-aware interventions, motivated by a basic principle in reinforcement learning: under the Bellman equation, the optimal policy selects actions that maximize the immediate reward plus the value function. When a decision maker follows a suboptimal policy, this policy-value consistency no longer holds, creating discrepancies between the actions taken by the policy and those that maximize the immediate reward plus the value of the next state. We show that these policy-value inconsistencies naturally identify opportunities for intervention. We formalize this problem in a Markov decision process where an AI assistant may override human actions under an intervention budget. In the single-intervention regime, we show that the optimal strategy is to recommend the action that maximizes the human value function. For settings with multiple interventions, we propose a tractable approximation that prioritizes interventions based on the magnitude of the policy-value discrepancy. We evaluate these ideas in the domain of chess by learning models of humans from large-scale gameplay data. In simulation, our approach consistently outperforms interventions based on the strongest chess engine (Stockfish) in a wide range of settings. A within-subject human study with 20 players and 600 games further shows that our interventions significantly improve performance for low- and mid-skill players while matching expert-engine interventions for high-skill players.

Problem

• human-AI collaboration
• sequential decision-making
• intervention strategy
• value-aware intervention
• policy-value inconsistency

Innovation

• value-aware interventions
• policy-value discrepancy
• human-AI collaboration
• sequential decision-making
• Markov decision process