Don't Reinvent the Wheel: Efficient Instruction-Following Text Embedding based on Guided Space Transformation

2025-05-30

AI Summary

To address the high computational cost of re-encoding entire corpora for each new instruction in instruction-aware text embedding, this paper proposes GSTransform—a lightweight Guided Space Transformation framework. Its core innovation lies in the first explicit disentanglement of instruction-invariant representations from instruction-relevant information within pre-trained embeddings. Leveraging learnable linear or nonlinear spatial transformation modules, GSTransform dynamically redirects existing embeddings to align with user-specified instructions using only a few instruction-labeled examples. By jointly optimizing instruction semantic alignment loss and few-shot supervision, it eliminates the need for full re-encoding. Evaluated across nine real-world datasets and three downstream task categories, GSTransform consistently outperforms state-of-the-art methods, achieving 6×–300× faster inference while maintaining or improving embedding quality.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Feature Construction/Reformulation
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In this work, we investigate an important task named instruction-following text embedding, which generates dynamic text embeddings that adapt to user instructions, highlighting specific attributes of text. Despite recent advancements, existing approaches suffer from significant computational overhead, as they require re-encoding the entire corpus for each new instruction. To address this challenge, we propose GSTransform, a novel instruction-following text embedding framework based on Guided Space Transformation. Our key observation is that instruction-relevant information is inherently encoded in generic embeddings but remains underutilized. Instead of repeatedly encoding the corpus for each instruction, GSTransform is a lightweight transformation mechanism that adapts pre-computed embeddings in real time to align with user instructions, guided by a small amount of text data with instruction-focused label annotation. We conduct extensive experiments on three instruction-awareness downstream tasks across nine real-world datasets, demonstrating that GSTransform improves instruction-following text embedding quality over state-of-the-art methods while achieving dramatic speedups of 6~300x in real-time processing on large-scale datasets. The source code is available at https://github.com/YingchaojieFeng/GSTransform.

Problem

• Dynamic text embeddings adapting to user instructions
• Reducing computational overhead in instruction-following tasks
• Real-time transformation of pre-computed embeddings

Innovation

• Lightweight transformation for pre-computed embeddings
• Real-time adaptation to user instructions
• Guided by instruction-focused label annotations