Reviewriter: AI-Generated Instructions For Peer Review Writing

2025-06-04

Workshop on Innovative Use of NLP for Building Educational Applications

AI Summary

This study addresses the limited capacity of learners to produce effective peer reviews in German writing instruction. To this end, we developed the first AI-driven peer review guidance tool specifically designed for educational writing tasks. Methodologically, we propose an adaptive AI instruction generation paradigm: German-GPT2 is fine-tuned on authentic student peer review corpora and integrated with instruction-based prompt engineering and a human-AI collaborative evaluation mechanism. Validation employs dual-track assessment—quantitative metrics (BLEU/ROUGE) and expert human evaluation. Empirical results from a pilot with 14 German learners demonstrate statistically significant improvements in review normativity and reflective depth, alongside high technology acceptance. Our core contributions are twofold: (1) the first education-oriented adaptive instruction generation framework for peer review support, and (2) empirical validation of generative AI’s feasibility and pedagogical efficacy in authentic language teaching contexts.

Technology Category

• Natural Language Processing: Generation
• Humans and AI: AI for Accessibility
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Large Language Models (LLMs) offer novel opportunities for educational applications that have the potential to transform traditional learning for students. Despite AI-enhanced applications having the potential to provide personalized learning experiences, more studies are needed on the design of generative AI systems and evidence for using them in real educational settings. In this paper, we design, implement and evaluate exttt{Reviewriter}, a novel tool to provide students with AI-generated instructions for writing peer reviews in German. Our study identifies three key aspects: a) we provide insights into student needs when writing peer reviews with generative models which we then use to develop a novel system to provide adaptive instructions b) we fine-tune three German language models on a selected corpus of 11,925 student-written peer review texts in German and choose German-GPT2 based on quantitative measures and human evaluation, and c) we evaluate our tool with fourteen students, revealing positive technology acceptance based on quantitative measures. Additionally, the qualitative feedback presents the benefits and limitations of generative AI in peer review writing.

Problem

• Designing AI-generated instructions for peer review writing in German
• Fine-tuning German language models on student-written peer reviews
• Evaluating generative AI tool acceptance in educational settings

Innovation

• Fine-tuned German language models
• AI-generated adaptive instructions
• Evaluated with student feedback