EmbGen: Teaching with Reassembled Corpora

2026-05-19

AI Summary

This work addresses the high cost and difficulty of acquiring high-quality instruction-response pairs for supervised fine-tuning of small language models in specialized domains. To tackle this challenge, the authors propose an embedding-driven synthetic data generation method that decomposes raw corpora into entity-description pairs, leverages embedding similarity to cluster and reconstruct cross-paragraph and cross-document semantic structures, and employs multi-strategy sampling—encompassing local, intra-cluster, and inter-cluster approaches—combined with domain-specific system prompts to generate diverse question-answer pairs. Experimental results demonstrate that, on the most semantically heterogeneous dataset, the method achieves substantial improvements in Binary Accuracy of 12.5% and 88.9% under token budgets of 5M and 20M, respectively, significantly outperforming the strongest existing baselines and effectively mitigating the homogenization commonly observed in synthetic data.

Abstract

Adapting small instruction-tuned models to specialized domains often relies on supervised fine-tuning (SFT) on curated instruction-response examples, which is expensive to collect at scale. Synthetic training examples generated by a teacher LLM from a domain corpus can reduce this cost, but existing pipelines can produce homogenized outputs and do not consistently capture cross-passage or cross-document dependencies. We introduce EmbGen, a synthetic data generation pipeline that decomposes a corpus into entity-description pairs, reassembles them using semantic structure inferred from embedding similarity, and then generates question-answer (QA) pairs via proximity, intra-cluster, and inter-cluster sampling with cluster-specialized system prompts. We evaluate EmbGen against EntiGraph, InstructLab and Knowledge-Instruct on three datasets of varied semantic heterogeneity, under fixed token budgets (5 and 20 million tokens). We use lexical overlap metrics, an LLM-as-a-judge rubric, and Binary Accuracy, a composed metric combining Factual Accuracy and Completeness for evaluation. EmbGen improves Binary Accuracy on the most heterogeneous dataset by 12.5% at 5M and 88.9% at 20M tokens budget, relative to the strongest baseline, while remaining competitive across other datasets with lower heterogeneity.

Problem

• supervised fine-tuning
• synthetic data generation
• cross-document dependencies
• domain adaptation
• instruction tuning

Innovation

• synthetic data generation
• embedding-based reassembly
• cluster-specialized prompting
• cross-document dependency
• instruction tuning