DySem: Uncovering Dynamic Semantic Components via Multilingual Consensus for Calculating Semantic Textual Similarity

2026-05-28

AI Summary

Current large language models compute semantic textual similarity using fixed-dimensional final-layer hidden states, which are susceptible to interference from generic knowledge and high-dimensional redundant noise. This work proposes DySem, a training-free framework that introduces a novel dynamic semantic dimension selection mechanism grounded in multilingual consistency. For each input text pair, DySem adaptively identifies semantically relevant components within the model’s internal representations, constructs a shared low-dimensional semantic subspace, and computes similarity within this subspace. By discarding static high-dimensional embeddings in favor of context-sensitive, compact representations, DySem significantly enhances both the accuracy and efficiency of similarity computation. The method consistently outperforms existing approaches across multiple large language models while operating at substantially reduced dimensions.

Abstract

Calculating semantic textual similarity is a foundational task in natural language processing. Current large language models (LLMs) based methods typically rely on extracting last-layer hidden states with fixed dimensions to compute similarity for every text pairs. We argue that this paradigm is suffer from two limitations: (i) The last hidden layer encodes more general knowledge rather than just semantic knowledge, making it suboptimal for semantic similarity computation; (ii) The hidden layer dimensions of LLMs are generally very large, which introduces some redundancy and noise for representing semantics. In this work, we propose DySem, a novel training-free framework that investigates more semantic-related internal components of LLMs via multilingual consensus, and shifts away from static representation spaces in favor of dynamic, sample-specific semantic dimensions by constructing text-dependent joint semantic set and computes similarity over this shared dimensional subset. Extensive experiments across various LLMs show that our method consistently outperforms recent baselines while maintaining lower dimensions for similarity calculation. The code is released at https://github.com/szu-tera/DySem.

Problem

• semantic textual similarity
• large language models
• hidden states
• semantic representation
• dimension redundancy

Innovation

• Dynamic Semantic Components
• Multilingual Consensus
• Semantic Textual Similarity
• Training-Free Framework
• Sample-Specific Representation