Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Models

2025-02-19

AI Summary

Current sentence embedding models heavily rely on large-scale human-annotated data, while mainstream LLM-based generation approaches neglect fine-grained semantic ranking information. To address this, we propose a latent-space-controllable sentence pair synthesis framework that introduces conditional latent-space guidance into the LLM generation process—explicitly modeling semantic distance ordering among sentences. Our method jointly optimizes contrastive learning, ordinal ranking loss, and LLM distillation-based fine-tuning objectives. Crucially, it requires only a small number of synthetically generated ranked sentence pairs. Evaluated across multiple semantic similarity and retrieval benchmarks, our approach achieves state-of-the-art performance, significantly outperforming unsupervised and weakly supervised baselines. It effectively alleviates the annotation bottleneck and enhances embedding discriminability without sacrificing efficiency or scalability.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Sentence embedding is essential for many NLP tasks, with contrastive learning methods achieving strong performance using annotated datasets like NLI. Yet, the reliance on manual labels limits scalability. Recent studies leverage large language models (LLMs) to generate sentence pairs, reducing annotation dependency. However, they overlook ranking information crucial for fine-grained semantic distinctions. To tackle this challenge, we propose a method for controlling the generation direction of LLMs in the latent space. Unlike unconstrained generation, the controlled approach ensures meaningful semantic divergence. Then, we refine exist sentence embedding model by integrating ranking information and semantic information. Experiments on multiple benchmarks demonstrate that our method achieves new SOTA performance with a modest cost in ranking sentence synthesis.

Problem

• Reducing dependency on manual annotation in NLP.
• Incorporating ranking information for semantic distinctions.
• Enhancing sentence embedding models using LLMs.

Innovation

• Leverages LLMs for sentence pair generation
• Controls LLM generation in latent space
• Integrates ranking and semantic information