H3D: Benchmarking Unsupervised Text Hashing for Fine-Grained Document Deduplication

2026-07-09

AI Summary

This work addresses the lack of a unified evaluation framework for unsupervised text hashing methods in fine-grained scientific document deduplication. We introduce the H3D benchmark, which systematically evaluates a range of non-learning-based approaches—such as MinHash, SimHash, and Winnowing—and semantic-aware hashing techniques leveraging frozen BGE embeddings combined with quantization strategies like BIHash and LSHash. For the first time, lexical/structural fingerprints and semantic representations are compared within a consistent framework. Our analysis reveals rank equivalences among different similarity measures in hash spaces and elucidates trade-offs among ranking quality, efficiency, and robustness: lexical methods excel in computational efficiency for near-duplicate detection, whereas semantic approaches better handle content rewording at higher computational cost. This study significantly enhances interpretability and reproducibility in text hashing research.

Abstract

Document hashing provides compact representations for efficient similarity search and document deduplication, but existing studies rarely compare hashing pipelines under a unified protocol for fine-grained scientific documents. H3D is an unsupervised text hashing benchmark for fine-grained document deduplication. It evaluates representative unsupervised non-learning hashing approaches (MinHash, SimHash, Winnowing, FuzzyHash, FlyHash) together with semantic-sensitive methods built from frozen BGE embeddings and two quantization strategies (BGE-BIHash and BGE-LSHash). The non-learning methods generate hash fingerprints through manually designed mathematical rules without training or labeled similarity pairs, which distinguishes them from neural semantic hashing models. We benchmark all methods on CSFCube and RELISH, two datasets that provide complementary evaluation settings: facet-level analysis for scientific-document similarity and larger-scale split-level evaluation for biomedical similarity search. H3D jointly reports ranking quality (MAP, NDCG@20), efficiency, and robustness under controlled text compression. The results show a consistent trade-off: lexical and structural fingerprints are competitive for near-duplicate matching, while semantic-sensitive representations better preserve similarity under content rewriting, at higher computational cost. We further analyze when different similarity measures become rank-equivalent for specific hash representations, improving the interpretability and reproducibility of method comparisons.

Problem

• unsupervised text hashing
• fine-grained document deduplication
• hashing benchmark
• scientific documents
• similarity search

Innovation

• unsupervised text hashing
• fine-grained document deduplication
• semantic-sensitive hashing
• hash benchmarking
• rank-equivalence analysis