🤖 AI Summary
This work addresses the pervasive anisotropy in the embedding spaces of mainstream language models, which violates the isotropy assumption underlying the Word Embedding Association Test (WEAT) and thereby distorts bias measurements. To mitigate this issue, the study introduces Zero-phase Component Analysis (ZCA) whitening as a geometric preprocessing step within the WEAT pipeline. ZCA minimally perturbs the original embeddings while transforming their covariance matrix into the identity matrix, thereby restoring isotropy. Experiments across ten standard WEAT test sets and seven language models demonstrate that ZCA substantially reduces anisotropy, altering the statistical significance of over 30% of WEAT results and, in some models, even improving semantic similarity performance. These findings confirm that ZCA enhances the consistency and reliability of bias assessments in word embeddings.
📝 Abstract
We propose Zero-phase Component Analysis (ZCA) whitening as a geometric pre-processing step for the Word Embedding Association Test (WEAT). WEAT is a bias measurement method widely used in both computational social science and AI fairness research. It relies on cosine similarity as a measure of semantic association, which assumes that the embedding space is approximately isotropic. However, prior work has reported that many widely used language models do not satisfy this assumption, raising concerns about the reliability of bias measurements. ZCA whitening transforms the covariance of the embedding space into the identity matrix while minimizing perturbation to the original vectors. This transformation restores the isotropy condition on which WEAT relies. We evaluate our approach on ten standard WEAT test suites and seven models spanning three architectural families, yielding 70 model-task combinations. The results show that ZCA whitening substantially reduces the anisotropy of the embedding spaces across all models. Particularly for highly anisotropic models, we further observe improvements on standard semantic similarity benchmarks, indicating that the calibrated space better captures semantic associations. After calibration, over 30% of WEAT results change significance status, and effect sizes shift in both directions depending on bias category. These shifts suggest that uncalibrated measurements may both overestimate and underestimate the associations encoded in the embedding space. These findings indicate that previously reported bias measurements in anisotropic embedding spaces should be interpreted with caution and may benefit from re-evaluation with calibrated methods. Our approach contributes to restoring the measurement foundation of WEAT across both computational social science and AI fairness research.