Score-Only Distillation for Compact Dense Retrieval

2026-07-13

AI Summary

This work addresses the high computational cost of deploying large embedding models for dense retrieval in online settings. The authors propose a distillation method that relies solely on the teacher model’s output score vectors—without requiring access to its hidden states—and integrates hard negative mining with a compact dual-encoder architecture. They further introduce a memory-efficient, row-wise centered PairMSE loss tailored to the matching retrieval protocol, effectively transferring the teacher’s ranking capability to a smaller student model. Evaluated on a fixed set of eight tasks, the approach closes up to 50% of the performance gap between the base and teacher models. The resulting 0.6B-parameter student model achieves a 4.7× speedup in query encoding and a 9.7× speedup in document encoding compared to the teacher.

Abstract

Large embedding models improve retrieval quality, but serving large encoders online is expensive. We study whether a compact retriever can learn teacher ranking behavior from score vectors without access to teacher hidden states. The student trains on rows built from ground-truth positives and negative candidates produced by our data generation pipeline; we evaluate student-teacher hard-negative mining separately as an extension. We use a row-centered score-vector objective, a memory-efficient implementation of uniform all-pairs PairMSE loss. On a fixed eight-task evaluation panel, our distillation protocol recovers up to 50\% of the base-to-teacher gap. The distilled 0.6B student is 4.7$\times$ faster for query encoding and 9.7$\times$ faster for document encoding than sequential online teacher fusion. External-transfer performance after distillation remains mixed, so our evidence supports compression of teacher rankings under matched retrieval protocols.

Problem

• dense retrieval
• model distillation
• score-only learning
• compact retriever
• ranking compression

Innovation

• score-only distillation
• dense retrieval
• knowledge distillation
• PairMSE loss
• compact retriever