🤖 AI Summary
This study addresses the limitations of general-purpose pretrained language models in effectively capturing the linguistic characteristics of second-language learners’ writing, which constrains the performance of automated essay scoring (AES). To mitigate this issue, the authors propose a domain-adaptive continued pretraining (DAPT) strategy that leverages CEFR-level filtering to align pretraining data with target proficiency levels. Specifically, three Transformer-based encoders are further pretrained on the EFCAMDAT corpus and evaluated on the FCE and IELTS datasets for both scoring accuracy and cross-dataset transferability. Experimental results demonstrate significant improvements in AES performance when the CEFR levels of the pretraining data match those of the downstream task—such as B1–B2 for FCE—whereas cross-dataset transfer yields inconsistent gains, underscoring the critical importance of proficiency-level alignment in model adaptation.
📝 Abstract
Recent automated essay scoring (AES) studies increasingly use pretrained transformer models, but these models are usually pretrained on general-domain English and may under-represent second-language learner writing. This study investigates whether domain-adaptive continued pretraining (DAPT) on the EFCAMDAT learner corpus improves transformer-based AES for English proficiency tests. We apply DAPT to three transformer encoders and evaluate them on FCE and IELTS in both in-domain scoring and few-shot cross-dataset transfer. Full-corpus DAPT produces mixed results across models, datasets, and metrics. Further analyses suggest that these mixed effects are partly explained by mismatches in proficiency, genre, and communicative purpose between EFCAMDAT and the downstream datasets. A proficiency-based ablation shows that targeted DAPT using CEFR-aligned subsets improves downstream scoring more reliably than full-corpus DAPT, especially for FCE with B1--B2 data. However, these gains do not consistently improve cross-dataset transfer. Overall, the findings suggest that continued pretraining on a learner-writing corpus can benefit in-domain AES for English assessment when the pretraining data is sufficiently aligned with the downstream assessment settings. However, it does not automatically improve transferability across different English proficiency test datasets.