SemHash-LLM: A Multi-Granularity Semantic Hashing Framework for Document Deduplication

2026-07-01

AI Summary

This work addresses the challenge of balancing semantic equivalence and computational efficiency in large-scale document deduplication by proposing a multi-granularity semantic hashing framework. The approach integrates character-, token-, and document-level signals through attention-weighted MinHash, semantic projection hashing, and a gated fusion mechanism to enable efficient representation. To enhance robustness against template contamination and short-text perturbations, it incorporates contrastive margin learning, uncertainty estimation, and knowledge distillation from large language models. Coupled with a cascaded filtering strategy, the method maintains high deduplication quality while constraining neural verification overhead to less than 1% of total computational cost.

Abstract

Large scale document deduplication must preserve semantic equivalence while remaining efficient over massive corpora. We present SemHash LLM, a multi granularity framework that unifies semantic projection hashing, attention weighted MinHash, contrastive boundary learning, and selective LLM based adjudication. The method combines character, token, and document level signals through gated fusion, then applies a cascaded filtering pipeline for efficient candidate reduction. Semantic projection hashing learns compact binary codes in distilled LLM embedding space, while attention weighted Min- Hash suppresses boilerplate and emphasizes informative content. Adaptive decision boundaries and uncertainty estimation further improve robustness across template pollution, short text perturbation, containment, and viral fragments. Experiments show that SemHash LLM achieves strong duplicate detection quality with less than one percent neural verification cost.

Problem

• document deduplication
• semantic equivalence
• large scale
• efficiency
• massive corpora

Innovation

• semantic hashing
• multi-granularity fusion
• attention-weighted MinHash
• contrastive boundary learning
• LLM-based adjudication