Maximum Inner Product is Query-Scaled Nearest Neighbor

2025-03-10

AI Summary

Maximum Inner Product Search (MIPS) in high-dimensional spaces faces two fundamental challenges: distortion introduced by spatial transformations when reducing MIPS to Nearest Neighbor Search (NNS), and the absence of a triangle inequality in the inner-product metric, which hinders effective pruning. This paper establishes, for the first time, a rigorous theoretical equivalence between MIPS and NNS directly in the original space—without any vector transformation. To address the concentration-of-norm issue caused by large-magnitude vectors, we propose the Spherical Path Proximity Graph (PSP), a novel graph structure operating on the unit sphere. PSP incorporates magnitude-aware edge pruning and an Adaptive Early Termination (AET) mechanism to significantly suppress redundant computations. Experiments demonstrate that our method achieves, on average, 35% faster query latency and reduces index size by 3× compared to state-of-the-art graph-based approaches. The solution has been deployed in Shopee’s production search system, and we release the first industrial-scale e-commerce MIPS benchmark dataset.

Technology Category

• Search and Optimization: Distributed Search
• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Intelligent Query Processing

Application Category

• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Social Networks and Social Media: Social mining and social search on the Web

Abstract

Maximum Inner Product Search (MIPS) for high-dimensional vectors is pivotal across databases, information retrieval, and artificial intelligence. Existing methods either reduce MIPS to Nearest Neighbor Search (NNS) while suffering from harmful vector space transformations, or attempt to tackle MIPS directly but struggle to mitigate redundant computations due to the absence of the triangle inequality. This paper presents a novel theoretical framework that equates MIPS with NNS without requiring space transformation, thereby allowing us to leverage advanced graph-based indices for NNS and efficient edge pruning strategies, significantly reducing unnecessary computations. Despite a strong baseline set by our theoretical analysis, we identify and address two persistent challenges to further refine our method: the introduction of the Proximity Graph with Spherical Pathway (PSP), designed to mitigate the issue of MIPS solutions clustering around large-norm vectors, and the implementation of Adaptive Early Termination (AET), which efficiently curtails the excessive exploration once an accuracy bottleneck is reached. Extensive experiments reveal the superiority of our method over existing state-of-the-art techniques in search efficiency, scalability, and practical applicability. Compared with state-of-the-art graph based methods, it achieves an average 35% speed-up in query processing and a 3x reduction in index size. Notably, our approach has been validated and deployed in the search engines of Shopee, a well-known online shopping platform. Our code and an industrial-scale dataset for offline evaluation will also be released to address the absence of e-commerce data in public benchmarks.

Problem

• Efficient high-dimensional vector search without space transformation.
• Reducing redundant computations in Maximum Inner Product Search.
• Improving search efficiency and scalability in practical applications.

Innovation

• Equates MIPS with NNS without space transformation
• Introduces Proximity Graph with Spherical Pathway
• Implements Adaptive Early Termination strategy