A perceptual bias of AI Logical Argumentation Ability in Writing

2025-11-27

AI Summary

This study investigates whether human prior beliefs systematically bias evaluations of AI’s logical reasoning capabilities. Employing a controlled experimental design combined with surveys, participants conducted double-blind assessments of logically structured arguments—some generated by AI and others authored by humans—rating their logical coherence and reporting attitudinal dispositions. Results reveal a significant cognitive bias: preexisting beliefs about AI capabilities substantially influence logicality ratings, independent of actual argument quality. Moreover, frequent AI users exhibit greater endorsement of AI-augmented independent reasoning and express less concern regarding AI’s potential to undermine human cognition. This work provides the first empirical evidence of an “AI capability perception bias”—a key psychological mechanism affecting human-AI interaction. By grounding findings in cognitive science, the study advances theoretical understanding of human-AI trust dynamics and offers actionable insights for designing more transparent, reliable, and cognitively aligned human-AI collaborative systems.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Can machines think? This is a central question in artificial intelligence research. However, there is a substantial divergence of views on the answer to this question. Why do people have such significant differences of opinion, even when they are observing the same real world performance of artificial intelligence? The ability of logical reasoning like humans is often used as a criterion to assess whether a machine can think. This study explores whether human biases influence evaluations of the reasoning abilities of AI. An experiment was conducted where participants assessed two texts on the same topic, one AI generated and one human written,to test for perceptual biases in evaluating logical reasoning. Based on the experimental findings, a questionnaire was designed to quantify the attitudes toward AI.The results reveal a bias in perception. The evaluations of the logical reasoning ability of AI generated texts are significantly influenced by the preconceived views on the logical reasoning abilities of AI. Furthermore, frequent AI users were less likely to believe that AI usage undermines independent thinking.This study highlights the need to address perceptual biases to improve public understanding of AI's capabilities and foster better human AI interactions.

Problem

• Investigates human perceptual biases in evaluating AI's logical reasoning ability
• Examines how preconceived views affect assessments of AI-generated text logic
• Explores the need to address biases for better public understanding of AI

Innovation

• Experiment comparing AI and human text evaluation
• Questionnaire quantifying attitudes towards AI reasoning
• Identifying bias in AI logical reasoning perception