AI evaluation may bias perceptions: The importance of context in interpreting academic writing

2026-05-26

AI Summary

This study addresses the systematic bias in current AI usage assessment methods, which often overlook contextual differences across countries and academic disciplines, leading to inaccurate estimations of AI involvement in scholarly writing. Leveraging large-scale journal publication data from Dimensions, the authors employ a large language model to rewrite human-authored abstracts and establish customized “AI similarity” baselines tailored to specific country–discipline combinations. This approach effectively disentangles inherent disciplinary and national writing styles from genuine AI-generated characteristics. The proposed contextualized benchmark substantially mitigates the distortions introduced by uniform thresholds— which tend to overestimate AI use in certain regions and fields while underestimating it in others—and demonstrates markedly fairer and more accurate evaluation performance for publications projected in 2025.

Abstract

This paper examines how estimates of AI use in scientific writing can be biased when evaluation methods ignore contextual differences across countries and fields. Using large-scale data on journal publications from Dimensions, we construct AI-likeness benchmarks based on differences between human-written and LLM-rephrased abstracts. We show that a pooled benchmark may confound pre-existing stylistic variation with AI-generated text, producing substantial distortions across country-field groups even in pre-LLM publications. In contrast, country-field-specific benchmarks attenuate such distortions and provide a more credible baseline for comparison. Applying these methods to publications in 2025 reveals that the pooled benchmark systematically overestimates AI use in certain countries and fields while underestimating it in others. These findings highlight the importance of context-aware measurement for accurate and equitable evaluation of AI use in science.

Problem

• AI evaluation
• context bias
• academic writing
• cross-country comparison
• field-specific variation

Innovation

• context-aware benchmarking
• AI detection
• scientific writing
• large language models
• cross-national comparison