Examining Student Interactions with a Pedagogical AI-Assistant for Essay Writing and their Impact on Students Writing Quality

2025-12-09

AI Summary

Prior research lacks empirical analysis of how pedagogical generative AI writing assistants (EWAs) interact with students during argumentative writing and how such interactions affect writing quality, particularly across instructional phases. Method: Analyzing 1,282 authentic interaction logs, we applied sequential pattern mining and K-means clustering to identify behavioral patterns and correlated them with human-rated writing scores. Contribution/Results: We identified two dominant user behaviors: (1) “active drafting”—submitting paragraphs for immediate feedback—strongly associated with higher overall writing quality; and (2) “outline-focused planning”—significantly linked to superior structural coherence (r = 0.36, p < 0.01). This study provides the first empirical evidence that an “active writing–instant feedback”闭环 outperforms passive querying. Based on these findings, we propose a design principle emphasizing behavioral scaffolding in AI writing tutors—specifically, prompting and reinforcing productive writing actions—to enhance pedagogical effectiveness. Our work bridges theoretical instructional design with data-driven interaction analysis, offering both conceptual grounding and actionable guidelines for optimizing intelligent writing instruction systems.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

The dynamic nature of interactions between students and GenAI, as well as their relationship to writing quality, remains underexplored. While most research has examined how general-purpose GenAI can support writing, fewer studies have investigated how students interact with pedagogically designed systems across different phases of the writing process. To address this gap, we evaluated a GenAI-driven essay-writing assistant (EWA) designed to support higher education students in argumentative writing. Drawing on 1,282 interaction logs from 32 undergraduates during a two-hour writing session, Sequential Pattern Mining and K-Means clustering were used to identify behavioral patterns. Two clusters emerged: Cluster 1 emphasized outline planning and essay structure, while Cluster 2 focused on content development. A Mann-Whitney U test revealed a moderate effect size (r = 0.36) in the essay Organization dimension, with Cluster 1 showing higher scores. Qualitative analysis indicated that students with better performance actively wrote and shared essay sections with EWA for feedback, rather than interacted passively by asking questions. These findings suggest implications for teaching and system design. Teachers can encourage active engagement, while future EWAs may integrate automatic labeling and monitoring to prompt students to move from questioning to writing, enabling fuller benefits from GenAI-supported learning.

Problem

• Examining student interactions with AI essay assistants
• Investigating impact of AI interactions on writing quality
• Identifying behavioral patterns in AI-assisted writing processes

Innovation

• Sequential Pattern Mining identifies student interaction patterns
• K-Means clustering groups students by writing behavior focus
• Automatic labeling and monitoring prompts active writing engagement