IMPARA-GED: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator

2025-06-03

AI Summary

Existing reference-free automatic grammatical error correction (GEC) evaluation methods suffer from weak grammatical error detection (GED) capability, resulting in low correlation with human judgments. This paper proposes the first unified framework for reference-free GEC quality assessment that deeply integrates GED: leveraging pretrained language models, it jointly optimizes multitask GED fine-tuning and semantic consistency modeling to enable synergistic evaluation of error localization and correction quality. Evaluated on the SEEDA meta-evaluation dataset, our method achieves the highest sentence-level correlation with human ratings (Pearson/Spearman), significantly outperforming all existing reference-free approaches. The core contribution lies in establishing fine-grained GED capability—not merely surface-level text similarity—as the foundational basis for assessment, thereby enhancing interpretability and robustness.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We propose IMPARA-GED, a novel reference-free automatic grammatical error correction (GEC) evaluation method with grammatical error detection (GED) capabilities. We focus on the quality estimator of IMPARA, an existing automatic GEC evaluation method, and construct that of IMPARA-GED using a pre-trained language model with enhanced GED capabilities. Experimental results on SEEDA, a meta-evaluation dataset for automatic GEC evaluation methods, demonstrate that IMPARA-GED achieves the highest correlation with human sentence-level evaluations.

Problem

• Enhancing grammatical error detection in GEC evaluation
• Developing reference-free automatic GEC quality assessment
• Improving correlation with human sentence-level evaluations

Innovation

• Reference-free GEC evaluation method
• Pre-trained language model enhanced
• Highest correlation with human evaluations