Gumbel Machine: Counterfactual Student Writing Generation via Gumbel Noise Steering

2026-05-26

AI Summary

This work addresses the challenge that existing methods for generating counterfactual examples of student writing often fail to simultaneously preserve fidelity to the original text and adherence to scoring rubrics, while also being limited to narrow domains. To overcome these limitations, the authors propose a modular framework that leverages the instruction-following capabilities of large language models together with a novel controllable decoding algorithm, β-Hindsight. By incorporating a tunable mechanism based on Gumbel noise, the approach explicitly regulates the similarity between generated counterfactuals and the original essays during decoding. Extensive experiments across multiple student writing datasets demonstrate that the method reliably produces high-quality counterfactual examples that are both highly similar to the source texts and strictly aligned with established scoring criteria, exhibiting strong generalizability and practical utility.

Abstract

An effective method of teaching across disciplines is to provide examples of high-quality work. However, an example may be significantly different from a student's current work, making it challenging for them to emulate. An ideal learning demonstration is a counterfactual version of the student work, an improved version that is still similar to their own. Existing automated approaches for counterfactual text generation using Large Language Models (LLMs) result in domain-specific systems that are difficult to translate into practical applications. We present the Gumbel Machine, a flexible, modular approach to generating counterfactuals that leverages LLM instruction-following capabilities while encouraging similarity to a reference factual text. Central to our approach is a novel, controlled decoding algorithm, $β$-Hindsight control, which uses latent randomness as a tunable similarity control mechanism during counterfactual generation. Experiments on datasets of student writing, scored on various criteria, demonstrate the effectiveness of our approach at generating counterfactuals both rubric-consistent and similar to a reference.

Problem

• counterfactual generation
• student writing
• text similarity
• instruction-following
• educational demonstration

Innovation

• Gumbel Machine
• counterfactual generation
• controlled decoding
• β-Hindsight control
• instruction-following LLMs