A survey of diversity quantification in natural language processing: The why, what, where and how

2025-07-28

AI Summary

Diversity in NLP lacks a unified theoretical foundation, resulting in fragmented metrics, inconsistent terminology, and poor cross-task comparability. Method: We propose the first systematic taxonomy of diversity for NLP, innovatively adapting the ecological and economic three-dimensional model—differentiation, evenness, and richness—to establish a formal theoretical basis. Building on Stirling’s (2007) framework, we conduct a systematic review and content analysis of ACL papers from the past six years containing “diversity” in their titles or abstracts. Contribution/Results: This yields a standardized classification scheme that rigorously addresses four core questions: *why* measure diversity, *what* aspects to measure, *where* (i.e., at which linguistic or model level) to measure, and *how* to operationalize measurement. Our framework substantially enhances theoretical coherence and methodological comparability across diversity metrics, providing a unified benchmark and a clear roadmap for advancing diversity research in NLP.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Learning Preferences or Rankings
• Application Domains: Humanities & Computational Social Science

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

The concept of diversity has received increased consideration in Natural Language Processing (NLP) in recent years. This is due to various motivations like promoting and inclusion, approximating human linguistic behavior, and increasing systems' performance. Diversity has however often been addressed in an ad hoc manner in NLP, and with few explicit links to other domains where this notion is better theorized. We survey articles in the ACL Anthology from the past 6 years, with "diversity" or "diverse" in their title. We find a wide range of settings in which diversity is quantified, often highly specialized and using inconsistent terminology. We put forward a unified taxonomy of why, what on, where, and how diversity is measured in NLP. Diversity measures are cast upon a unified framework from ecology and economy (Stirling, 2007) with 3 dimensions of diversity: variety, balance and disparity. We discuss the trends which emerge due to this systematized approach. We believe that this study paves the way towards a better formalization of diversity in NLP, which should bring a better understanding of this notion and a better comparability between various approaches.

Problem

• Surveying diversity quantification in NLP to unify inconsistent terminology.
• Proposing a taxonomy for why, what, where, and how diversity is measured.
• Applying a framework from ecology and economy to standardize diversity measures.

Innovation

• Surveying ACL Anthology for diversity quantification
• Proposing unified taxonomy for diversity measurement
• Applying ecological framework with three diversity dimensions