DistillER: Knowledge Distillation in Entity Resolution with Large Language Models

2026-02-05

AI Summary

This work proposes the first unsupervised knowledge distillation framework tailored for entity resolution, addressing the challenge of high computational costs associated with large language models (LLMs) and the reliance of existing approaches on expensive models or manual annotations. By leveraging an LLM as a teacher, the framework systematically explores data selection, knowledge extraction, and distillation strategies to effectively transfer the teacher’s capabilities to a lightweight student model—without requiring ground-truth labels. Experimental results demonstrate that the proposed method significantly outperforms both supervised and unsupervised baselines across multiple benchmarks, achieving high accuracy while substantially improving inference efficiency. Moreover, the distilled student model retains the ability to generate high-quality explanations, offering both performance and interpretability advantages.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Natural Language Processing: Information Extraction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Recent advances in Entity Resolution (ER) have leveraged Large Language Models (LLMs), achieving strong performance but at the cost of substantial computational resources or high financial overhead. Existing LLM-based ER approaches operate either in unsupervised settings and rely on very large and costly models, or in supervised settings and require ground-truth annotations, leaving a critical gap between time efficiency and effectiveness. To make LLM-powered ER more practical, we investigate Knowledge Distillation (KD) as a means to transfer knowledge from large, effective models (Teachers) to smaller, more efficient models (Students) without requiring gold labels. We introduce DistillER, the first framework that systematically bridges this gap across three dimensions: (i) Data Selection, where we study strategies for identifying informative subsets of data; (ii) Knowledge Elicitation, where we compare single- and multi-teacher settings across LLMs and smaller language models (SLMs); and (iii) Distillation Algorithms, where we evaluate supervised fine-tuning and reinforcement learning approaches. Our experiments reveal that supervised fine-tuning of Students on noisy labels generated by LLM Teachers consistently outperforms alternative KD strategies, while also enabling high-quality explanation generation. Finally, we benchmark DistillER against established supervised and unsupervised ER methods based on LLMs and SLMs, demonstrating significant improvements in both effectiveness and efficiency.

Problem

• Entity Resolution
• Large Language Models
• Knowledge Distillation
• Efficiency
• Label-free

Innovation

• Knowledge Distillation
• Entity Resolution
• Large Language Models
• Label-Free Learning
• Model Compression