Understanding the Influence of Synthetic Data for Text Embedders

2025-09-07

AI Summary

The absence of publicly available, reproducible, high-quality synthetic datasets hinders systematic investigation into how synthetic data affects the generalization of text embedding models. Method: This work introduces the first large-scale, LLM-generated synthetic dataset released openly, coupled with a unified contrastive learning training framework and comprehensive multi-task downstream evaluation (including retrieval, clustering, and classification). Contribution/Results: Empirical analysis reveals that gains from synthetic data exhibit pronounced task locality—improvements are sparse and highly imbalanced across tasks. Significant performance trade-offs exist across tasks, with degradation observed in several cases. These findings expose fundamental limitations of current synthetic-data approaches for universal embedding modeling, challenging the prevailing assumption that synthetic data universally enhances robustness. The study provides critical empirical evidence to inform future data construction methodologies and evaluation paradigms in text representation learning.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recent progress in developing general purpose text embedders has been driven by training on ever-growing corpora of synthetic LLM-generated data. Nonetheless, no publicly available synthetic dataset exists, posing a barrier to studying its role for generalization. To address this issue, we first reproduce and publicly release the synthetic data proposed by Wang et al. (Mistral-E5). Our synthetic data is high quality and leads to consistent improvements in performance. Next, we critically examine where exactly synthetic data improves model generalization. Our analysis reveals that benefits from synthetic data are sparse and highly localized to individual datasets. Moreover, we observe trade-offs between the performance on different categories and data that benefits one task, degrades performance on another. Our findings highlight the limitations of current synthetic data approaches for building general-purpose embedders and challenge the notion that training on synthetic data leads to more robust embedding models across tasks.

Problem

• Lack of public synthetic data for text embedder studies
• Understanding where synthetic data improves model generalization
• Trade-offs in performance across different task categories

Innovation

• Reproduce and release synthetic data
• Analyze synthetic data's localized benefits
• Reveal trade-offs in task performance