Reevaluation of Large Neighborhood Search for MAPF: Findings and Opportunities

2024-07-12

AI Summary

This work addresses three critical issues in evaluating Large Neighborhood Search (LNS) methods for Anytime Multi-Agent Path Finding (MAPF): inaccurate baseline performance, inconsistent evaluation metrics, and non-reproducible implementations of learning-based approaches. To this end, we introduce the first open-source, standardized LNS evaluation framework for MAPF. Through systematic reproduction and fair comparison of state-of-the-art LNS algorithms, we establish that rule-based heuristics—particularly CBS-LNS—serve as strong, robust baselines; existing supervised learning methods show no significant improvement in solution quality or runtime efficiency. Based on these findings, we propose three novel research directions: (1) handling high-latency agents, (2) context-aware re-planning, and (3) dynamic neighborhood size adaptation. Extensive experiments validate the robustness of rule-based baselines. We publicly release all code, trained models, and benchmark datasets to enable reproducible and comparable MAPF-LNS research.

Technology Category

• Multiagent Systems: Multiagent Planning
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling
• Search and Optimization: Heuristic Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Multi-Agent Path Finding (MAPF) aims to arrange collision-free goal-reaching paths for a group of agents. Anytime MAPF solvers based on large neighborhood search (LNS) have gained prominence recently due to their flexibility and scalability, leading to a surge of methods, especially those leveraging machine learning, to enhance neighborhood selection. However, several pitfalls exist and hinder a comprehensive evaluation of these new methods, which mainly include: 1) Lower than actual or incorrect baseline performance; 2) Lack of a unified evaluation setting and criterion; 3) Lack of a codebase or executable model for supervised learning methods. To address these challenges, we introduce a unified evaluation framework, implement prior methods, and conduct an extensive comparison of prominent methods. Our evaluation reveals that rule-based heuristics serve as strong baselines, while current learning-based methods show no clear advantage on time efficiency or improvement capacity. Our extensive analysis also opens up new research opportunities for improving MAPF-LNS, such as targeting high-delayed agents, applying contextual algorithms, optimizing replan order and neighborhood size, where machine learning can potentially be integrated. Code and data are available at https://github.com/ChristinaTan0704/mapf-lns-unified.

Problem

• Evaluating pitfalls in MAPF-LNS methods' comprehensive assessment
• Addressing inconsistent baselines and evaluation standards in MAPF-LNS
• Assessing learning-based methods' inefficiency in MAPF-LNS optimization

Innovation

• Unified evaluation framework for MAPF-LNS methods
• Rule-based heuristics as strong baselines
• Machine learning integration opportunities identified