A Quality Diversity Method to Automatically Generate Multi-Agent Path Finding Benchmark Maps

2024-09-10

AI Summary

Existing MAPF benchmarks rely on hand-crafted maps, leading to biased evaluation, algorithmic overfitting, and unfair comparisons. To address these limitations, this work introduces the Quality-Diversity (QD) paradigm to MAPF benchmark construction for the first time. We propose an automated map generation framework integrating MAP-Elites with Neural Cellular Automata (NCA), capable of synthesizing diverse, task-relevant, interpretable, and challenging map corpora with rich structural semantics. Our approach systematically covers critical failure modes—including congestion, deadlocks, and long-path dependencies—beyond the constraints of manual design. Extensive experiments across multiple MAPF paradigms (search-based, priority-based, rule-based, and learning-based algorithms) demonstrate that the generated maps significantly improve evaluation comprehensiveness, fairness, and reproducibility, while enabling fine-grained failure attribution analysis.

Technology Category

• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Search and Optimization: Sampling/Simulation-based Search
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

We use the Quality Diversity (QD) algorithm with Neural Cellular Automata (NCA) to generate benchmark maps for Multi-Agent Path Finding (MAPF) algorithms. Previously, MAPF algorithms are tested using fixed, human-designed benchmark maps. However, such fixed benchmark maps have several problems. First, these maps may not cover all the potential failure scenarios for the algorithms. Second, when comparing different algorithms, fixed benchmark maps may introduce bias leading to unfair comparisons between algorithms. Third, since researchers test new algorithms on a small set of fixed benchmark maps, the design of the algorithms may overfit to the small set of maps. In this work, we take advantage of the QD algorithm to (1) generate maps with patterns to comprehensively understand the performance of MAPF algorithms, (2) be able to make fair comparisons between two MAPF algorithms, providing further information on the selection between two algorithms and on the design of the algorithms. Empirically, we employ this technique to generate diverse benchmark maps to evaluate and compare the behavior of different types of MAPF algorithms, including search-based, priority-based, rule-based, and learning-based algorithms. Through both single-algorithm experiments and comparisons between algorithms, we identify patterns where each algorithm excels and detect disparities in runtime or success rates between different algorithms.

Problem

• Multi-Agent Path Finding
• Algorithm Evaluation
• Map Dependency

Innovation

• QD Algorithm
• NCA
• Diverse Map Generation