Training Sparse Mixture Of Experts Text Embedding Models

2025-02-11

AI Summary

To address the deployment bottlenecks—namely high inference latency and excessive memory consumption—posed by large-parameter Transformer-based text embedding models in retrieval-augmented generation (RAG) scenarios, this work pioneers the integration of sparse Mixture-of-Experts (MoE) architecture into general-purpose text embedding. We propose Nomic Embed v2, the first open-source, general-purpose sparse MoE text embedding model. Built upon a Transformer encoder, it employs gated routing and multi-task joint training—including contrastive learning—to achieve substantial reductions in memory footprint and latency at equivalent parameter counts. Empirically, Nomic Embed v2 surpasses comparable models across diverse monolingual and multilingual benchmarks (e.g., MIRACL, BEIR), demonstrating superior cross-lingual robustness. Notably, its performance matches that of dense models twice its size. All code, model weights, and evaluation datasets are fully open-sourced.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Transformer-based text embedding models have improved their performance on benchmarks like MIRACL and BEIR by increasing their parameter counts. However, this scaling approach introduces significant deployment challenges, including increased inference latency and memory usage. These challenges are particularly severe in retrieval-augmented generation (RAG) applications, where large models' increased memory requirements constrain dataset ingestion capacity, and their higher latency directly impacts query-time performance. While causal language models have addressed similar efficiency challenges using Mixture of Experts (MoE) architectures, this approach hasn't been successfully adapted to the general text embedding setting. In this paper, we introduce Nomic Embed v2, the first general purpose MoE text embedding model. Our model outperforms models in the same parameter class on both monolingual and multilingual benchmarks while also maintaining competitive performance with models twice its size. We open-source all code, models, and evaluation data to ensure full reproducibility of our training pipeline.

Problem

• Addresses deployment challenges of large text embedding models.
• Improves efficiency in retrieval-augmented generation applications.
• Introduces first MoE-based general purpose text embedding model.

Innovation

• Sparse Mixture of Experts
• Transformer-based text embedding
• Open-source training pipeline