A visual observation on the geometry of UMAP projections of the difference vectors of antonym and synonym word pair embeddings

2026-03-25

AI Summary

This study investigates the geometric distinctions between antonym and synonym pairs in word embeddings, with a focus on whether their difference vectors encode a discernible structural signature of "antonymy." By computing difference vectors for such word pairs across multiple mainstream embedding models and visualizing them via UMAP dimensionality reduction, the research reveals that antonym pairs consistently exhibit a stable, vortex-like geometric pattern in low-dimensional space—a configuration markedly distinct from that of synonym pairs and robust across models. These findings suggest that semantic opposition may possess a universal geometric representation within word embedding spaces, offering a novel perspective on the internal semantic organization of these models.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Cognitive Modeling & Cognitive Systems: Analogy
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Antonyms, or opposites, are sometimes defined as \emph{word pairs that have all of the same contextually relevant properties but one}. Seeing how transformer models seem to encode concepts as directions, this begs the question if one can detect ``antonymity'' in the geometry of the embedding vectors of word pairs, especially based on their difference vectors. Such geometrical studies are then naturally contrasted by comparing antonymic pairs to their opposites; synonyms. This paper started as an exploratory project on the complexity of the systems needed to detect the geometry of the embedding vectors of antonymic word pairs. What we now report is a curious ``swirl'' that appears across embedding models in a somewhat specific projection configuration.

Problem

• antonyms
• synonyms
• word embeddings
• vector geometry
• UMAP projections

Innovation

• UMAP projection
• difference vectors
• antonym geometry
• word embeddings
• semantic directions