Rethinking Word Similarity: Semantic Similarity through Classification Confusion

2025-02-08

AI Summary

Traditional cosine similarity between static word embeddings fails to capture the context-dependency, asymmetry, and polysemy inherent in semantic similarity. To address this, we propose Word Confusion—a novel metric that reframes word similarity as the confusion probability in a contextual embedding classification task, where dynamically selected confusable words serve as semantic features. This constitutes the first paradigm shift from distance-based metrics to classification-based confusion modeling. Our method integrates the Tversky cognitive model with contextualized embeddings (e.g., BERT), defining similarity via off-diagonal probabilities in the confusion matrix—enabling interpretable, dimension-aware semantic measurement and diachronic tracking. Evaluated on MEN, WordSim353, and SimLex benchmarks, Word Confusion matches or exceeds cosine-similarity baselines. Empirically, it reveals a systematic 18th-century semantic shift of the French word *révolution*—from “popular action” to “state-led action”—demonstrating its capacity for fine-grained historical semantic analysis.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Cognitive Modeling & Cognitive Systems: Analogy
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• Web Mining and Content Analysis: Models for Web evolution
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Word similarity has many applications to social science and cultural analytics tasks like measuring meaning change over time and making sense of contested terms. Yet traditional similarity methods based on cosine similarity between word embeddings cannot capture the context-dependent, asymmetrical, polysemous nature of semantic similarity. We propose a new measure of similarity, Word Confusion, that reframes semantic similarity in terms of feature-based classification confusion. Word Confusion is inspired by Tversky's suggestion that similarity features be chosen dynamically. Here we train a classifier to map contextual embeddings to word identities and use the classifier confusion (the probability of choosing a confounding word c instead of the correct target word t) as a measure of the similarity of c and t. The set of potential confounding words acts as the chosen features. Our method is comparable to cosine similarity in matching human similarity judgments across several datasets (MEN, WirdSim353, and SimLex), and can measure similarity using predetermined features of interest. We demonstrate our model's ability to make use of dynamic features by applying it to test a hypothesis about changes in the 18th C. meaning of the French word"revolution"from popular to state action during the French Revolution. We hope this reimagining of semantic similarity will inspire the development of new tools that better capture the multi-faceted and dynamic nature of language, advancing the fields of computational social science and cultural analytics and beyond.

Problem

• Measure semantic similarity dynamically
• Improve context-dependent word similarity
• Test historical meaning changes

Innovation

• Feature-based classification confusion
• Dynamic similarity features selection
• Contextual embeddings to word identities