BitNet Text Embeddings

2026-06-24

AI Summary

This work addresses the challenges of slow inference and high storage and bandwidth costs associated with high-dimensional, full-precision text embeddings generated by large language models in large-scale retrieval. The authors propose BITEMBED, a framework that achieves end-to-end ultra-low-bit text embeddings for the first time by converting pretrained large language models into BitNet-style ternary-weight encoders with quantized activations. Representation quality is enhanced through contrastive learning–based continual pretraining and supervised fine-tuning. A novel dual knowledge distillation mechanism—leveraging both similarity distributions and attention relations—is introduced to support multi-precision embedding outputs, enabling flexible adaptation to varying resource constraints. Evaluated on the MMTEB (eng, v2) benchmark, models based on Qwen3-0.6B and Gemma3-270M achieve performance close to that of full-precision teacher models while substantially reducing storage and bandwidth requirements.

Abstract

LLM-based text embedders have substantially improved retrieval and semantic representation quality, but their deployment remains costly: large backbone models slow down embedding inference, while high-dimensional full-precision embeddings impose substantial storage and bandwidth overhead on large-scale indexes. In this paper, we present BITEMBED, an extreme low-bit framework for LLM-based text embedding that jointly targets encoding efficiency and vector storage. BITEMBED converts pretrained LLM backbones into BitNet-style embedding encoders with ternary weights, quantized activations, and lightweight normalization refinement. The converted model is adapted to representation learning through continual contrastive pre-training, followed by supervised contrastive fine-tuning with both similarity-distribution distillation and attention-relation distillation from a full-precision teacher. Beyond quantizing the backbone, BITEMBED further trains output embeddings to support multiple storage precisions meeting different storage needs in various scenarios. Experiments on MMTEB (eng, v2) with Qwen3-0.6B and Gemma3-270M show that BITEMBED is largely comparable to full precision teacher embedders. Moreover, BITEMBED flexibly obtains text embeddings of various precisions, achieving a trade-off between performance and storage cost.

Problem

• text embedding
• large language models
• storage overhead
• inference latency
• embedding quantization

Innovation

• low-bit embedding
• BitNet
• quantization
• contrastive learning
• embedding distillation