A Robust and Efficient Pipeline for Enterprise-Level Large-Scale Entity Resolution

2025-08-04

AI Summary

Enterprise-scale entity resolution (ER) faces dual challenges of scalability and accuracy under high-concurrency, massive-data conditions. This paper proposes MERAI—an AI-powered, end-to-end entity matching pipeline integrating adaptive blocking, hardware-aware inverted indexing, and a lightweight semantic similarity model—to jointly optimize precision and memory efficiency. Compared to state-of-the-art systems Dedupe and Splink, MERAI achieves up to an 8.2% F1-score improvement and 3.1× higher throughput on datasets containing up to 15.7 million records, while demonstrating strong robustness and near-linear scalability. To our knowledge, this is the first work to co-design learnable blocking strategies with hardware-optimized indexing for large-scale ER. MERAI establishes a new paradigm for industrial data integration—delivering high accuracy, computational efficiency, deployment readiness, and operational stability in production environments.

Technology Category

• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Machine Learning: Hardware-aware ML
• Search and Optimization: Distributed Search

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Entity resolution (ER) remains a significant challenge in data management, especially when dealing with large datasets. This paper introduces MERAI (Massive Entity Resolution using AI), a robust and efficient pipeline designed to address record deduplication and linkage issues in high-volume datasets at an enterprise level. The pipeline's resilience and accuracy have been validated through various large-scale record deduplication and linkage projects. To evaluate MERAI's performance, we compared it with two well-known entity resolution libraries, Dedupe and Splink. While Dedupe failed to scale beyond 2 million records due to memory constraints, MERAI successfully processed datasets of up to 15.7 million records and produced accurate results across all experiments. Experimental data demonstrates that MERAI outperforms both baseline systems in terms of matching accuracy, with consistently higher F1 scores in both deduplication and record linkage tasks. MERAI offers a scalable and reliable solution for enterprise-level large-scale entity resolution, ensuring data integrity and consistency in real-world applications.

Problem

• Addressing large-scale entity resolution challenges efficiently
• Improving record deduplication and linkage accuracy in enterprises
• Scaling ER solutions beyond memory constraints for big datasets

Innovation

• AI-powered pipeline for large-scale entity resolution
• Handles datasets up to 15.7 million records
• Outperforms Dedupe and Splink in accuracy