Reproducibility in the Control of Autonomous Mobility-on-Demand Systems

2025-06-09

AI Summary

The AMoD (Autonomous Mobility-on-Demand) field suffers from a lack of standardized evaluation and reproducible research practices; opaque modeling assumptions, experimental setups, and algorithm implementations severely hinder scientific validation and progress. To address this, we systematically identify critical non-reproducible elements across the full pipeline—modeling, control, simulation, algorithm design, and evaluation. We propose the first structured reproducibility assessment framework tailored to AMoD, introducing a novel “Reproducibility Checklist.” Grounded in systems engineering and research meta-analysis, our approach integrates systematic literature review, process modeling, and multi-platform empirical validation to derive generalizable best practices. The resulting standardized guidelines substantially enhance experimental transparency, result credibility, and cross-study comparability in AMoD—and by extension, in related connected and autonomous systems—gaining broad community endorsement.

Technology Category

• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Application Domains: Mobility, Driving & Flight
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Autonomous Mobility-on-Demand (AMoD) systems, powered by advances in robotics, control, and Machine Learning (ML), offer a promising paradigm for future urban transportation. AMoD offers fast and personalized travel services by leveraging centralized control of autonomous vehicle fleets to optimize operations and enhance service performance. However, the rapid growth of this field has outpaced the development of standardized practices for evaluating and reporting results, leading to significant challenges in reproducibility. As AMoD control algorithms become increasingly complex and data-driven, a lack of transparency in modeling assumptions, experimental setups, and algorithmic implementation hinders scientific progress and undermines confidence in the results. This paper presents a systematic study of reproducibility in AMoD research. We identify key components across the research pipeline, spanning system modeling, control problems, simulation design, algorithm specification, and evaluation, and analyze common sources of irreproducibility. We survey prevalent practices in the literature, highlight gaps, and propose a structured framework to assess and improve reproducibility. Specifically, concrete guidelines are offered, along with a"reproducibility checklist", to support future work in achieving replicable, comparable, and extensible results. While focused on AMoD, the principles and practices we advocate generalize to a broader class of cyber-physical systems that rely on networked autonomy and data-driven control. This work aims to lay the foundation for a more transparent and reproducible research culture in the design and deployment of intelligent mobility systems.

Problem

• Lack of standardized evaluation practices in AMoD systems
• Challenges in reproducibility due to opaque methodologies
• Need for transparent frameworks in autonomous mobility research

Innovation

• Systematic study of AMoD reproducibility challenges
• Proposes structured framework with reproducibility checklist
• Guidelines for replicable cyber-physical systems research