Argument Rarity-based Originality Assessment for AI-Assisted Writing

2026-02-02

AI Summary

In the context of widespread adoption of large language models, traditional writing assessments centered on quality struggle to effectively capture students’ critical thinking and originality of ideas. This work proposes the Argument Rarity-based Originality Assessment (AROA) framework, which redefines originality as rarity within a corpus and decouples it from quality as an independent evaluation dimension. AROA integrates four components—rarity of structure, claim, evidence, and cognitive depth—using density estimation and a quality-adjustment mechanism to form a unified automatic scoring system. Empirical results reveal a trade-off between quality and originality: high-quality texts often feature more common claims. Furthermore, while AI-generated texts exhibit structural complexity comparable to human writing, their claim rarity is significantly lower, underscoring a fundamental limitation in content originality.

Technology Category

• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Knowledge Representation and Reasoning: Argumentation
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

As Large Language Models (LLMs) have become capable of effortlessly generating high-quality text, traditional quality-focused writing assessment is losing its significance. If the essential goal of education is to foster critical thinking and original perspectives, assessment must also shift its paradigm from quality to originality. This study proposes Argument Rarity-based Originality Assessment (AROA), a framework for automatically evaluating argumentative originality in student essays. AROA defines originality as rarity within a reference corpus and evaluates it through four complementary components: structural rarity, claim rarity, evidence rarity, and cognitive depth. The framework quantifies the rarity of each component using density estimation and integrates them with a quality adjustment mechanism, thereby treating quality and originality as independent evaluation axes. Experiments using human essays and AI-generated essays revealed a strong negative correlation between quality and claim rarity, demonstrating a quality-originality trade-off where higher-quality texts tend to rely on typical claim patterns. Furthermore, while AI essays achieved comparable levels of structural complexity to human essays, their claim rarity was substantially lower than that of humans, indicating that LLMs can reproduce the form of argumentation but have limitations in the originality of content.

Problem

• originality assessment
• AI-assisted writing
• argument rarity
• large language models
• writing evaluation

Innovation

• Argument Rarity
• Originality Assessment
• Large Language Models
• Density Estimation
• Quality-Originality Trade-off