KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model

2025-01-02

AI Summary

Existing general-purpose embedding models suffer from weak multilingual generalization due to noisy and insufficiently diverse training data. To address this, we focus on data quality enhancement and propose three innovations: (1) persona-guided controllable synthetic data generation; (2) ranking-consistency-driven dynamic sample filtering; and (3) semi-isomorphic, task-aware batch sampling. We further pioneer the adaptation of the Qwen2-0.5B autoregressive language model—previously unexplored for embedding tasks—to construct a lightweight (<1B parameters) multilingual embedding framework. This framework integrates LLM distillation, dynamic filtering, and task-aware training. Evaluated on the MTEB multilingual benchmark, our model comprehensively outperforms all existing models of comparable scale, establishing a new state-of-the-art for sub-1B-parameter embedding models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As retrieval-augmented generation prevails in large language models, embedding models are becoming increasingly crucial. Despite the growing number of general embedding models, prior work often overlooks the critical role of training data quality. In this work, we introduce KaLM-Embedding, a general multilingual embedding model that leverages a large quantity of cleaner, more diverse, and domain-specific training data. Our model has been trained with key techniques proven to enhance performance: (1) persona-based synthetic data to create diversified examples distilled from LLMs, (2) ranking consistency filtering to remove less informative samples, and (3) semi-homogeneous task batch sampling to improve training efficacy. Departing from traditional BERT-like architectures, we adopt Qwen2-0.5B as the pre-trained model, facilitating the adaptation of auto-regressive language models for general embedding tasks. Extensive evaluations of the MTEB benchmark across multiple languages show that our model outperforms others of comparable size, setting a new standard for multilingual embedding models with<1B parameters.

Problem

• Generic Embedding Models
• Training Data Quality
• Model Performance Limitations

Innovation

• KaLM-Embedding
• Qwen2-0.5B
• MTEB Benchmark