Can Large Language Models Differentiate Harmful from Argumentative Essays? Steps Toward Ethical Essay Scoring

2026-01-09

International Conference on Computational Linguistics

AI Summary

This study addresses a critical gap in automated essay scoring (AES) systems and large language models (LLMs), which often fail to detect harmful content—such as racism or gender bias—in argumentative essays and may erroneously assign high scores to such texts. To tackle this issue, the work introduces an ethical dimension into AES evaluation for the first time, presenting the Harmful Essay Detection (HED) benchmark dataset specifically designed to assess the ability to identify harmful arguments. It also proposes a corresponding ethical evaluation framework. Experimental results demonstrate that prevailing LLMs and AES systems generally lack the capacity to distinguish between harmful and legitimate reasoning, revealing a significant blind spot in their moral judgment. This research establishes a foundational benchmark and offers a clear direction for developing ethically aware next-generation AES systems.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization

Abstract

This study addresses critical gaps in Automated Essay Scoring (AES) systems and Large Language Models (LLMs) with regard to their ability to effectively identify and score harmful essays. Despite advancements in AES technology, current models often overlook ethically and morally problematic elements within essays, erroneously assigning high scores to essays that may propagate harmful opinions. In this study, we introduce the Harmful Essay Detection (HED) benchmark, which includes essays integrating sensitive topics such as racism and gender bias, to test the efficacy of various LLMs in recognizing and scoring harmful content. Our findings reveal that: (1) LLMs require further enhancement to accurately distinguish between harmful and argumentative essays, and (2) both current AES models and LLMs fail to consider the ethical dimensions of content during scoring. The study underscores the need for developing more robust AES systems that are sensitive to the ethical implications of the content they are scoring.

Problem

• Automated Essay Scoring
• Large Language Models
• Harmful Content Detection
• Ethical Scoring
• Argumentative Essays

Innovation

• Harmful Essay Detection
• Automated Essay Scoring
• Large Language Models
• Ethical AI
• Content Moderation