Bidding Games on Markov Decision Processes with Quantitative Reachability Objectives

2024-12-27

AI Summary

This work introduces auction-based two-player stochastic games on Markov decision processes (MDPs), where a reachability player and a safety player alternately bid to determine state transitions—the former seeks to maximize the probability of reaching a target state, while the latter aims to minimize it. We are the first to incorporate auction mechanisms into MDP-based game frameworks, proposing a “budget–reachability-probability” threshold relation—replacing the scalar thresholds used in classical graph-based games—and developing the first value-iteration framework applicable to general MDPs. Our method integrates quantitative reachability analysis, reduction to simple stochastic games, and exact algorithms for acyclic MDPs, enabling both approximate computation of the threshold relation and exact solutions for acyclic cases. We prove that threshold decision complexity is at least as hard as solving simple stochastic games. This work establishes a novel modeling paradigm and algorithmic foundation for multi-agent stochastic games.

Technology Category

• Game Theory and Economic Paradigms: Auctions and Market-Based Systems
• Multiagent Systems: Mechanism Design
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

Graph games are fundamental in strategic reasoning of multi-agent systems and their environments. We study a new family of graph games which combine stochastic environmental uncertainties and auction-based interactions among the agents, formalized as bidding games on (finite) Markov decision processes (MDP). Normally, on MDPs, a single decision-maker chooses a sequence of actions, producing a probability distribution over infinite paths. In bidding games on MDPs, two players -- called the reachability and safety players -- bid for the privilege of choosing the next action at each step. The reachability player's goal is to maximize the probability of reaching a target vertex, whereas the safety player's goal is to minimize it. These games generalize traditional bidding games on graphs, and the existing analysis techniques do not extend. For instance, the central property of traditional bidding games is the existence of a threshold budget, which is a necessary and sufficient budget to guarantee winning for the reachability player. For MDPs, the threshold becomes a relation between the budgets and probabilities of reaching the target. We devise value-iteration algorithms that approximate thresholds and optimal policies for general MDPs, and compute the exact solutions for acyclic MDPs, and show that finding thresholds is at least as hard as solving simple-stochastic games.

Problem

• Markov Decision Processes
• Auction Games
• Optimal Strategy

Innovation

• Auction Game Model
• Markov Decision Process
• Optimal Strategy Algorithm