How Large Language Models Are Changing MOOC Essay Answers: A Comparison of Pre- and Post-LLM Responses

2025-04-17

AI Summary

This study investigates the real-world impact of ChatGPT’s release on student writing behavior in online education. Using a longitudinal, temporally complete dataset of student essays from a massive open online course (MOOC) on AI ethics, we employ dynamic topic modeling (DTM), quantitative analysis of text length and stylistic features, keyword frequency tracking, and a pre-post quasi-experimental design to systematically compare writing patterns before and after ChatGPT’s launch. Results show a significant increase in average essay length, growing stylistic homogenization, and a sharp rise in AI/LLM-related terminology—while the thematic distribution of core ethical concepts remains stable. To our knowledge, this is the first empirical study leveraging authentic, continuous MOOC writing data to reveal the structural effects of large language models (LLMs) on learning outcomes. It provides critical evidence and methodological foundations for rethinking educational assessment and academic integrity governance in the AI era.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Philosophy and Ethics of AI: Artificial General Intelligence

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The release of ChatGPT in late 2022 caused a flurry of activity and concern in the academic and educational communities. Some see the tool's ability to generate human-like text that passes at least cursory inspections for factual accuracy ``often enough'' a golden age of information retrieval and computer-assisted learning. Some, on the other hand, worry the tool may lead to unprecedented levels of academic dishonesty and cheating. In this work, we quantify some of the effects of the emergence of Large Language Models (LLMs) on online education by analyzing a multi-year dataset of student essay responses from a free university-level MOOC on AI ethics. Our dataset includes essays submitted both before and after ChatGPT's release. We find that the launch of ChatGPT coincided with significant changes in both the length and style of student essays, mirroring observations in other contexts such as academic publishing. We also observe -- as expected based on related public discourse -- changes in prevalence of key content words related to AI and LLMs, but not necessarily the general themes or topics discussed in the student essays as identified through (dynamic) topic modeling.

Problem

• Analyzing LLM impact on MOOC essay length and style changes
• Assessing ChatGPT's influence on academic dishonesty in online education
• Comparing pre- and post-LLM student essay content trends

Innovation

• Analyzing pre- and post-ChatGPT MOOC essay responses
• Using dynamic topic modeling for theme identification
• Quantifying LLM impact on essay length and style