Quantifying Misattribution Unfairness in Authorship Attribution

2025-06-02

AI Summary

Misattribution errors in high-stakes forensic applications may lead to unjust scrutiny, yet existing evaluation paradigms overlook disparities in misattribution risk across authors. This work formally defines and quantifies author-level misattribution unfairness, introducing the Misattribution Unfairness Index (MAUIₖ)—a metric capturing the deviation in probability that non-authors are erroneously ranked among the top-k candidates for a given text. Leveraging geometric analysis of embedding spaces, rank-based statistics, and cross-model and cross-dataset benchmarking, we demonstrate that state-of-the-art authorship attribution models exhibit pervasive unfairness. Crucially, misattribution risk is strongly negatively correlated with the distance of an author’s embedding from the centroid of the embedding space: authors whose representations lie closer to the centroid face significantly higher false-attribution risk. These findings provide empirically grounded, interpretable insights to guide model calibration and user-facing risk communication.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Computer Vision: Bias, Fairness & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Authorship misattribution can have profound consequences in real life. In forensic settings simply being considered as one of the potential authors of an evidential piece of text or communication can result in undesirable scrutiny. This raises a fairness question: Is every author in the candidate pool at equal risk of misattribution? Standard evaluation measures for authorship attribution systems do not explicitly account for this notion of fairness. We introduce a simple measure, Misattribution Unfairness Index (MAUIk), which is based on how often authors are ranked in the top k for texts they did not write. Using this measure we quantify the unfairness of five models on two different datasets. All models exhibit high levels of unfairness with increased risks for some authors. Furthermore, we find that this unfairness relates to how the models embed the authors as vectors in the latent search space. In particular, we observe that the risk of misattribution is higher for authors closer to the centroid (or center) of the embedded authors in the haystack. These results indicate the potential for harm and the need for communicating with and calibrating end users on misattribution risk when building and providing such models for downstream use.

Problem

• Measure unfairness in authorship misattribution risks
• Evaluate models using Misattribution Unfairness Index (MAUIk)
• Analyze misattribution risks related to author embedding positions

Innovation

• Introduces Misattribution Unfairness Index (MAUIk)
• Quantifies unfairness in authorship attribution models
• Links unfairness to author embedding spatial distribution