Encode Errors: Representational Retrieval of In-Context Demonstrations for Multilingual Grammatical Error Correction

2026-06-13

AI Summary

This work addresses the limitations of large language models in few-shot grammatical error correction, which stem from their difficulty in retrieving examples that reflect error patterns rather than semantic similarity. The study reveals for the first time that internal model states encode grammatical error information and introduces Grammatical Error Representation (GER)—a novel, semantically neutral representation rich in error-specific features. Building upon GER, the authors develop a multilingual context-aware example retrieval mechanism. Their approach enables an 8B open-source model to match the performance of Deepseek2.5 and GPT-4o-mini on high-resource languages, while achieving up to a 1.20× improvement in F₀.₅ score over baselines for low-resource languages, substantially enhancing correction accuracy, efficiency, and interpretability.

Abstract

Grammatical Error Correction (GEC) involves detecting and correcting the wrong usage of grammar. While large language models (LLMs) with in-context learning (ICL) capabilities have shown significant progress on various natural language processing (NLP) tasks, their few-shot performance on GEC remains suboptimal. This is mainly due to the challenge of retrieving suitable in-context demonstrations that capture error patterns instead of semantic similarity. In this paper, we demonstrate that LLMs can inherently capture information related to grammatical errors through their internal states. From these states, we extract the Grammatical Error Representation (GER), an informative and semantically neutral encoding of grammatical errors. Our novel GER-based retrieval method significantly boosts performance in ICL settings on multilingual GEC datasets, improving the precision of correction. For high-resource languages, our results on 8B-sized open-source models match those of closed-source models such as Deepseek2.5 and GPT-4o-mini. For low-resource languages, our $F_{0.5}$ scores surpass the baseline by up to a factor of 1.20. This method provides a more precise and resource-efficient solution for multilingual GEC, offering a promising direction for interpretable GEC research.

Problem

• Grammatical Error Correction
• In-Context Learning
• Multilingual
• Demonstration Retrieval
• Few-Shot Learning

Innovation

• Grammatical Error Representation
• In-Context Learning
• Multilingual Grammatical Error Correction
• Error Pattern Retrieval
• Large Language Models