Dispersion Measures as Predictors of Lexical Decision Time, Word Familiarity, and Lexical Complexity

2025-01-11

AI Summary

This study systematically evaluates the cross-linguistic predictive validity of term dispersion metrics for lexical processing time, familiarity, and complexity. Using corpora from five languages, we employ multivariate linear regression and granularity-controlled experiments to compare log-range, log-frequency, and established dispersion measures (e.g., DP, VMR). Results demonstrate that log-range significantly outperforms log-frequency and consistently surpasses all other dispersion metrics across languages and tasks; as a complementary predictor to word frequency, it exhibits the strongest explanatory power. These findings indicate that simple, interpretable dispersion metrics—not complex computational models—possess superior psycholinguistic validity. Log-range thus provides a unified account of previously inconsistent findings regarding dispersion effects, resolving theoretical contradictions in the literature. By offering a robust, theoretically grounded metric, this work advances both empirical research on lexical representation and computational modeling of lexical access.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Various measures of dispersion have been proposed to paint a fuller picture of a word's distribution in a corpus, but only little has been done to validate them externally. We evaluate a wide range of dispersion measures as predictors of lexical decision time, word familiarity, and lexical complexity in five diverse languages. We find that the logarithm of range is not only a better predictor than log-frequency across all tasks and languages, but that it is also the most powerful additional variable to log-frequency, consistently outperforming the more complex dispersion measures. We discuss the effects of corpus part granularity and logarithmic transformation, shedding light on contradictory results of previous studies.

Problem

• Lexical Processing Time
• Linguistic Complexity
• Cross-linguistic Analysis

Innovation

• Log Range
• Predictive Accuracy
• Corpus Detail