Optimal Bounds for Open Addressing Without Reordering

2025-01-04

AI Summary

This work addresses the problem of optimizing lookup efficiency in open-addressing hash tables under the constraint that element reordering is prohibited. To overcome the fundamental bottleneck—namely, the inability of conventional uniform hashing to reduce probe counts—we propose a novel framework for constructing and composing hash functions, thereby refuting Yao’s long-standing conjecture on the optimality of uniform hashing. Theoretically, we establish tight upper and lower bounds on the expected and worst-case probe counts, and rigorously prove their matching via probabilistic analysis and information-theoretic arguments. Practically, our construction achieves significantly lower expected search complexity than all prior open-addressing schemes satisfying the no-reordering constraint. Our results yield the first complete solution for constrained hash table design that simultaneously attains theoretical optimality and constructive feasibility.

Technology Category

• Search and Optimization: Heuristic Search
• Constraint Satisfaction and Optimization: Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Tracking, profiling, and countermeasures against them

Abstract

In this paper, we revisit one of the simplest problems in data structures: the task of inserting elements into an open-addressed hash table so that elements can later be retrieved with as few probes as possible. We show that, even without reordering elements over time, it is possible to construct a hash table that achieves far better expected search complexities (both amortized and worst-case) than were previously thought possible. Along the way, we disprove the central conjecture left by Yao in his seminal paper ``Uniform Hashing is Optimal''. All of our results come with matching lower bounds.

Problem

• Open Addressing
• Hash Tables
• Probe Optimization

Innovation

• Open Addressing Hashing
• Optimized Retrieval Efficiency
• Refutation of Yao's Assumption