An Empirical Study to Understand How Students Use ChatGPT for Writing Essays

2025-01-17

AI Summary

This study investigates how middle school students use ChatGPT in argumentative writing and examines its effects on independent thinking, writing competence, self-efficacy, attitudes toward AI, and essay quality. Method: A controlled online experiment (n = 70) was conducted using a custom-built interactive platform that logged and coded all user queries; multivariate regression analyses integrated validated psychometric scales with computational text analysis. Contribution/Results: The study identifies— for the first time—five empirically grounded, behaviorally distinct AI usage patterns. Gender, race, and self-efficacy significantly predict pattern adoption (p < 0.05). Usage patterns differentially affect writing engagement (p < 0.01) and perceived authorship ownership (r = 0.42). Findings provide behavioral-log–based empirical evidence and a taxonomy-informed intervention framework to guide AI-integrated writing pedagogy.

Technology Category

• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Humans and AI: User Experience and Usability
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

As large language models (LLMs) advance and become widespread, students increasingly turn to systems like ChatGPT for assistance with writing tasks. Educators are concerned with students' usage of ChatGPT beyond cheating; using ChatGPT may reduce their critical engagement with writing, hindering students' learning processes. The negative or positive impact of using LLM-powered tools for writing will depend on how students use them; however, how students use ChatGPT remains largely unknown, resulting in a limited understanding of its impact on learning. To better understand how students use these tools, we conducted an online study $(n=70)$ where students were given an essay-writing task using a custom platform we developed to capture the queries they made to ChatGPT. To characterize their ChatGPT usage, we categorized each of the queries students made to ChatGPT. We then analyzed the relationship between ChatGPT usage and a variety of other metrics, including students' self-perception, attitudes towards AI, and the resulting essay itself. We found that factors such as gender, race, and perceived self-efficacy can help predict different AI usage patterns. Additionally, we found that different usage patterns were associated with varying levels of enjoyment and perceived ownership over the essay. The results of this study contribute to discussions about how writing education should incorporate generative AI-powered tools in the classroom.

Problem

• ChatGPT
• Essay Writing
• Impact on Students

Innovation

• ChatGPT
• Educational AI Applications
• Writing Quality Impact