Down with the Hierarchy: The 'H' in HNSW Stands for "Hubs"

2024-12-02

arXiv.org

AI Summary

This work challenges the necessity of hierarchical structure in HNSW for high-dimensional approximate nearest neighbor (ANN) search, proposing the “Hub Highway Hypothesis”: hub nodes naturally form efficient navigation paths that obviate hierarchical routing. To test this, we introduce Flat-NSW—a flattened navigable small-world graph devoid of explicit hierarchy—built via hub-aware edge construction. We conduct extensive ANN benchmarking across multiple real-world datasets and perform rigorous hubness analysis. Empirical results demonstrate that Flat-NSW achieves query latency and recall on par with HNSW, while reducing memory footprint by 30–50% on average. Our analysis further reveals that hub nodes are inherently prevalent in real high-dimensional data and spontaneously organize into high-speed retrieval pathways. These findings establish a de-hierarchical paradigm for graph-based indexing and provide theoretical grounding for hub-centric design principles in ANN search.

Technology Category

• Search and Optimization: Heuristic Search
• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Social Networks and Social Media: Social mining and social search on the Web

Abstract

Driven by recent breakthrough advances in neural representation learning, approximate near-neighbor (ANN) search over vector embeddings has emerged as a critical computational workload. With the introduction of the seminal Hierarchical Navigable Small World (HNSW) algorithm, graph-based indexes have established themselves as the overwhelmingly dominant paradigm for efficient and scalable ANN search. As the name suggests, HNSW searches a layered hierarchical graph to quickly identify neighborhoods of similar points to a given query vector. But is this hierarchy even necessary? A rigorous experimental analysis to answer this question would provide valuable insights into the nature of algorithm design for ANN search and motivate directions for future work in this increasingly crucial domain. To that end, we conduct an extensive benchmarking study covering more large-scale datasets than prior investigations of this question. We ultimately find that a flat navigable small world graph graph retains all of the benefits of HNSW on high-dimensional datasets, with latency and recall performance essentially emph{identical} to the original algorithm but with less memory overhead. Furthermore, we go a step further and study emph{why} the hierarchy of HNSW provides no benefit in high dimensions, hypothesizing that navigable small world graphs contain a well-connected, frequently traversed ``highway"of hub nodes that maintain the same purported function as the hierarchical layers. We present compelling empirical evidence that the emph{Hub Highway Hypothesis} holds for real datasets and investigate the mechanisms by which the highway forms. The implications of this hypothesis may also provide future research directions in developing enhancements to graph-based ANN search.

Problem

• High-Dimensional Vector Spaces
• Approximate Nearest Neighbor Search
• Memory-Efficient Algorithms

Innovation

• Similarity Search
• Memory Efficiency
• Hub Highway Points