A Reinforcement Learning Approach for Dynamic Rebalancing in Bike-Sharing System

2024-02-05

arXiv.org

AI Summary

To address station imbalance (overfull/empty stations) in bike-sharing systems caused by stochastic demand, this paper proposes a spatiotemporal-coordinated multi-agent reinforcement learning (MARL) framework for dynamic rebalancing. We formulate the problem as a continuous-time multi-agent Markov decision process (CT-MAMDP), overcoming limitations of conventional discrete-time synchronous scheduling. The model jointly captures vehicle-level asynchronous decision-making, spatiotemporal demand dynamics, and exogenous factors (e.g., weather and time-of-day), and employs a deep Q-network (DQN) architecture to enable cooperative optimization across heterogeneous agents. Extensive experiments on historical data across multiple realistic scenarios demonstrate that our method significantly reduces load-miss rates—outperforming a multi-period mixed-integer programming baseline. Furthermore, the trained model supports real-time, low-latency dispatch decisions and exhibits strong practical deployability.

Technology Category

• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Multiagent Systems: Multiagent Learning
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Bike-Sharing Systems provide eco-friendly urban mobility, contributing to the alleviation of traffic congestion and to healthier lifestyles. Efficiently operating such systems and maintaining high customer satisfaction is challenging due to the stochastic nature of trip demand, leading to full or empty stations. Devising effective rebalancing strategies using vehicles to redistribute bikes among stations is therefore of uttermost importance for operators. As a promising alternative to classical mathematical optimization, reinforcement learning is gaining ground to solve sequential decision-making problems. This paper introduces a spatio-temporal reinforcement learning algorithm for the dynamic rebalancing problem with multiple vehicles. We first formulate the problem as a Multi-agent Markov Decision Process in a continuous time framework. This allows for independent and cooperative vehicle rebalancing, eliminating the impractical restriction of time-discretized models where vehicle departures are synchronized. A comprehensive simulator under the first-arrive-first-serve rule is then developed to facilitate the learning process by computing immediate rewards under diverse demand scenarios. To estimate the value function and learn the rebalancing policy, various Deep Q-Network configurations are tested, minimizing the lost demand. Experiments are carried out on various datasets generated from historical data, affected by both temporal and weather factors. The proposed algorithms outperform benchmarks, including a multi-period Mixed-Integer Programming model, in terms of lost demand. Once trained, it yields immediate decisions, making it suitable for real-time applications. Our work offers practical insights for operators and enriches the integration of reinforcement learning into dynamic rebalancing problems, paving the way for more intelligent and robust urban mobility solutions.

Problem

• Developing reinforcement learning algorithms for dynamic bike rebalancing with multiple vehicles
• Comparing single-policy and dual-policy approaches to optimize inventory and routing decisions
• Addressing station imbalances caused by stochastic demand patterns in bike-sharing systems

Innovation

• Uses single deep Q-network for joint decisions
• Decouples inventory and routing into dual policies
• Models rebalancing as continuous-time Markov process