The Effect of Idea Elaboration on the Automatic Assessment of Idea Originality

2026-04-22

AI Summary

This study addresses a critical limitation in automated evaluation systems for creative tasks: their tendency to exhibit self-preference bias by overrating AI-generated content that aligns with their own stylistic patterns. By comparing the scoring consistency of fine-tuned large language models (OCSAI, CLAUS), prompt-driven ChatGPT-4o, and rigorously trained human raters on divergent thinking tasks, the research identifies idea elaboration as a pivotal factor influencing originality judgments. Crucially, when elaboration level is controlled, the models’ preferential bias toward AI-generated responses markedly diminishes. This finding offers both a novel theoretical insight and empirical foundation for developing more equitable and reliable frameworks for automated creativity assessment.

Technology Category

• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Learning Preferences or Rankings

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Automatic systems are increasingly used to assess the originality of responses in creative tasks. They offer a potential solution to key limitations of human assessment (cost, fatigue, and subjectivity), but there is preliminary evidence of a self-preference bias. Accordingly, automatic systems tend to prefer outcomes that are more closely related to their style, rather than to the human one. In this paper, we investigated how Large Language Models (LLMs) align with human raters in assessing the originality of responses in a divergent thinking task. We analysed 4,813 responses to the Alternate Uses Task produced by higher and lower creative humans and ChatGPT-4o. Human raters were two university students who underwent intensive training. Machine raters were two specialised systems fine-tuned on AUT responses and corresponding human ratings (OCSAI and CLAUS) and ChatGPT-4o, which was prompted with the same instructions as human raters. Results confirmed the presence of a self-preference bias in LLMs. Automatic systems tended to privilege artificial responses. However, this self-preference bias disappeared when the analyses controlled for the idea elaboration. We discuss theoretical and methodological implications of these findings by highlighting future directions for research on creativity assessment.

Problem

• originality assessment
• self-preference bias
• Large Language Models
• divergent thinking
• automatic creativity evaluation

Innovation

• self-preference bias
• idea elaboration
• originality assessment
• Large Language Models
• divergent thinking