Machine-assisted writing evaluation: Exploring pre-trained language models in analyzing argumentative moves

2025-03-25

AI Summary

Manual annotation of argumentative moves in longitudinal academic writing is costly and impedes large-scale developmental analysis. Method: This study develops a BERT-based automated system to identify six argumentative moves (e.g., claim, evidence, counterargument) across 1,643 longitudinal essays by Chinese EFL learners, introducing a human–model collaborative annotation framework. Contribution/Results: It represents the first systematic application of pretrained language models (PLMs) to longitudinal modeling of argumentative structure development and writing quality prediction. The system achieves an overall F1-score of 0.743—significantly outperforming prior approaches—and demonstrates strong validity as a substitute for manual coding. PLM-derived annotations accurately capture learners’ cognitive progression from unilateral claims toward multi-perspective argumentation and robustly differentiate low-, medium-, and high-quality writing.

Technology Category

• Natural Language Processing: Discourse, Pragmatics & Argument Mining
• Knowledge Representation and Reasoning: Argumentation
• Multiagent Systems: Agreement, Argumentation & Negotiation

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Studies of user behavior, including longitudinal effects of personalized systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The study investigates the efficacy of pre-trained language models (PLMs) in analyzing argumentative moves in a longitudinal learner corpus. Prior studies on argumentative moves often rely on qualitative analysis and manual coding, limiting their efficiency and generalizability. The study aims to: 1) to assess the reliability of PLMs in analyzing argumentative moves; 2) to utilize PLM-generated annotations to illustrate developmental patterns and predict writing quality. A longitudinal corpus of 1643 argumentative texts from 235 English learners in China is collected and annotated into six move types: claim, data, counter-claim, counter-data, rebuttal, and non-argument. The corpus is divided into training, validation, and application sets annotated by human experts and PLMs. We use BERT as one of the implementations of PLMs. The results indicate a robust reliability of PLMs in analyzing argumentative moves, with an overall F1 score of 0.743, surpassing existing models in the field. Additionally, PLM-labeled argumentative moves effectively capture developmental patterns and predict writing quality. Over time, students exhibit an increase in the use of data and counter-claims and a decrease in non-argument moves. While low-quality texts are characterized by a predominant use of claims and data supporting only oneside position, mid- and high-quality texts demonstrate an integrative perspective with a higher ratio of counter-claims, counter-data, and rebuttals. This study underscores the transformative potential of integrating artificial intelligence into language education, enhancing the efficiency and accuracy of evaluating students' writing. The successful application of PLMs can catalyze the development of educational technology, promoting a more data-driven and personalized learning environment that supports diverse educational needs.

Problem

• Assess PLM reliability in analyzing argumentative moves
• Utilize PLM annotations to track writing development patterns
• Predict writing quality using PLM-labeled argumentative moves

Innovation

• Using BERT for argumentative move analysis
• PLMs surpass existing models with F1 0.743
• AI enhances writing evaluation efficiency