Accurate but Natural? Diagnosing Grammatical and Idiomatic Gaps in Japanese EFL Writing

2026-08-10

AI Summary

This study addresses a key limitation in automated scoring systems for English writing by Japanese junior high school students, where grammatical accuracy and idiomaticity are often conflated. It proposes the first two-dimensional instructional diagnostic framework that disentangles these constructs. By implementing a hierarchical large language model correction pipeline, the approach separately generates literal error corrections and idiomatic rewrites. Coupled with a regular expression–driven CEFR-J grammar extractor, it quantifies distinct gaps in grammatical accuracy and idiomaticity. Empirical analysis reveals critical challenges—such as omissions of definite articles and third-person singular -s—as well as idiomaticity deviations, including underuse of -ing forms and would, and overreliance on can and SVO structures. The framework effectively differentiates performance errors, structural avoidance, and L1 transfer, offering both theoretical insights and practical tools for targeted pedagogical intervention.

Abstract

Second language writing research distinguishes grammatical accuracy from native-like idiomaticity, yet automated writing evaluation often conflates these dimensions. This study introduces a layered LLM-correction pipeline that isolates structural errors from unnaturalness by generating literal error corrections and idiomatic revisions for 3,830 English writing samples from 120 Japanese junior high school students. Applying the regex-based CEFR-J grammar extractor, we quantify two diagnostic measures: accuracy gaps (structures attempted but incorrectly produced) and idiomatic gaps (grammatically correct structures underused or overused relative to native norms). Results reveal distinct patterns: definite articles, third-person singular -s, and modals (would, could) exhibit significant accuracy difficulties, while -ing forms and hypothetical modals (would) show the largest idiomatic underuse, with simple present verbs, subject-verb-object patterns, and modal can conversely exhibiting the most pronounced overuse. A two-dimensional instructional typology maps error rates against idiomatic gaps, distinguishing accurate but overused grammar items from error-prone or avoided complex forms requiring targeted production practice. This framework advances pedagogical feedback by enabling teachers to diagnose whether learner difficulties arise from inaccurate execution, structural avoidance, or L1-mapped overreliance, supporting evidence-based interventions tailored to the specific needs of each learner.

Problem

• grammatical accuracy
• idiomaticity
• EFL writing
• Japanese learners
• writing diagnostics

Innovation

• layered LLM-correction pipeline
• accuracy gaps
• idiomatic gaps
• CEFR-J grammar extractor
• two-dimensional instructional typology