On the Emergence of Linear Analogies in Word Embeddings

2025-05-24

AI Summary

The theoretical origin of linear analogical structures (e.g., king − man + woman ≈ queen) in word embeddings has remained elusive. Method: We propose the first analytically tractable generative model for binary semantic attributes, integrating matrix spectral analysis, logarithmic transformation, and co-occurrence statistics to formalize how analogies emerge from the spectral properties of co-occurrence probability matrices and latent semantic attribute generation. Contribution/Results: We theoretically prove that analogical structure robustly emerges under dimensionality reduction, log-transformation, and data sparsification. The model precisely reproduces the saturation dynamics of analogy performance, achieving high alignment between theoretical predictions and empirical results on Wiki corpora and the Mikolov benchmark (mean absolute error < 1.2%). This work establishes the first unified, verifiable, generative explanation for the analogical capability of word vectors.

Technology Category

• Cognitive Modeling & Cognitive Systems: Analogy
• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Models such as Word2Vec and GloVe construct word embeddings based on the co-occurrence probability $P(i,j)$ of words $i$ and $j$ in text corpora. The resulting vectors $W_i$ not only group semantically similar words but also exhibit a striking linear analogy structure -- for example, $W_{ ext{king}} - W_{ ext{man}} + W_{ ext{woman}} approx W_{ ext{queen}}$ -- whose theoretical origin remains unclear. Previous observations indicate that this analogy structure: (i) already emerges in the top eigenvectors of the matrix $M(i,j) = P(i,j)/P(i)P(j)$, (ii) strengthens and then saturates as more eigenvectors of $M (i, j)$, which controls the dimension of the embeddings, are included, (iii) is enhanced when using $log M(i,j)$ rather than $M(i,j)$, and (iv) persists even when all word pairs involved in a specific analogy relation (e.g., king-queen, man-woman) are removed from the corpus. To explain these phenomena, we introduce a theoretical generative model in which words are defined by binary semantic attributes, and co-occurrence probabilities are derived from attribute-based interactions. This model analytically reproduces the emergence of linear analogy structure and naturally accounts for properties (i)-(iv). It can be viewed as giving fine-grained resolution into the role of each additional embedding dimension. It is robust to various forms of noise and agrees well with co-occurrence statistics measured on Wikipedia and the analogy benchmark introduced by Mikolov et al.

Problem

• Explains linear analogy structure in word embeddings
• Analyzes co-occurrence probability's role in embeddings
• Proposes generative model for semantic attribute interactions

Innovation

• Uses co-occurrence probabilities for word embeddings
• Introduces theoretical generative model with attributes
• Explains linear analogy structure in embeddings