Labeling Training Data for Entity Matching Using Large Language Models

2026-06-27

AI Summary

This work addresses the trade-off between annotation cost and inference efficiency in entity matching by proposing a knowledge distillation–based auto-labeling framework. While traditional approaches rely heavily on expensive human-annotated data and large language models (LLMs) offer label-free inference at high computational cost, the proposed method leverages powerful LLMs (e.g., GPT-5.2) to automatically generate labels for candidate entity pairs. These labels are refined through optimized candidate selection and post-processing strategies to train lightweight student models such as Ditto or RoBERTa. Evaluated on five benchmark datasets, the approach achieves performance comparable to fully human-annotated models, with an F1 score gap of no more than 2%, while reducing annotation costs to just \$28–\$41—saving approximately 470 hours of manual labeling—and accelerating inference by 41.5× to 534×, thereby effectively balancing accuracy and efficiency.

Abstract

Recent large language models (LLMs) achieve strong performance on entity matching without requiring task-specific training data. However, applying these models to large sets of candidate pairs remains slow and costly. In contrast, entity matchers using traditional machine learning methods or small language models (SLMs), such as RoBERTa, offer much faster inference but require task-specific training data. This paper investigates whether the need to provide task-specific training data can be avoided by using knowledge-distillation workflows, in which an LLM serves as a teacher model to label training pairs that are subsequently used to train a smaller student model. We investigate knowledge distillation for entity matching along the following dimensions: pair-selection strategy, teacher model, label post-processing method, and student model. We evaluate the workflows using the Abt-Buy, Walmart-Amazon, WDC Products, DBLP-ACM, and DBLP-Scholar benchmarks, and compare the performance of student models trained with machine-labeled data to the performance of the same models trained using the benchmark training sets. Our experiments show that student models trained using the machine-labeled sets perform approximately on par with models trained on the benchmark training sets, with the remaining differences in both directions staying below two F1 points. Using GPT-5.2 to label the training sets for all five benchmarks costs US\$28.31 to US\$40.88, whereas manually labeling the same training sets is estimated to require 470 hours of work. At inference time, Ditto is 41.5 to 534 times faster than directly using an LLM to perform the matching tasks. These results indicate that current LLMs, when combined with a suitable pair-selection method, can substantially reduce or even eliminate the manual effort required to label use case-specific training data for entity matching.

Problem

• Entity Matching
• Training Data Labeling
• Knowledge Distillation
• Large Language Models
• Manual Annotation

Innovation

• knowledge distillation
• entity matching
• large language models
• automated labeling
• student-teacher learning