On Parallel External-Memory Bidirectional Search (Extended Abstract)

2024-06-01

Symposium on Combinatorial Search

AI Summary

To address the search efficiency bottleneck on ultra-large-scale graphs, this paper proposes the first parallel and external-memory-cooperative bidirectional best-first search framework. Unlike conventional single-direction external-memory (PEM) algorithms, our approach systematically integrates bidirectional search into the PEM architecture. We design a generic task-scheduling mechanism and a memory-aware bidirectional expansion strategy to jointly optimize I/O efficiency and load balancing. Experimental evaluation on billion-node graphs demonstrates near-linear parallel speedup, over 40% reduction in memory footprint, and significantly improved scalability and practicality of bidirectional search for ultra-large-scale graph processing.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Machine Learning: Graph-based Machine Learning

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Social Networks and Social Media: Social mining and social search on the Web

Abstract

Parallelization and External Memory (PEM) techniques significantly enhance the capabilities of search algorithms for solving large-scale problems. While previous research on PEM has primarily centered on unidirectional algorithms, this work presents a versatile PEM framework that integrates both uni- and bi-directional best-first search algorithms.

Problem

• Parallelization
• External Memory
• Bidirectional Search

Innovation

• Parallel External Memory
• Bidirectional Search
• PEM-BAE* Algorithm