Label Over Logic? How Source Cues Bias Human Fallacy Judgments More Than LLMs

2026-05-28

AI Summary

This study investigates whether human judgments of logical fallacies are influenced by source labels (e.g., “human” or “AI”) and whether large language models (LLMs) exhibit greater judgmental stability. Through a large-scale online experiment (N = 505) and comparative evaluations with state-of-the-art LLMs—including GPT-5.2, Gemini 2.5 Flash, and Claude Sonnet 4.5—the research provides the first systematic comparison of human and LLM reasoning performance under source-label interference. The findings reveal that human participants are significantly biased by source labels, particularly overrating content attributed to humans or human–AI collaboration. In contrast, LLMs demonstrate markedly stronger source-invariant logical reasoning, highlighting their potential to mitigate cognitive biases in evaluative tasks.

Abstract

As AI-generated and AI-assisted content floods online spaces, source labels attached to such content can distort human reasoning judgments, with downstream consequences for moderation, evaluation, and decision-making. Whether LLMs share this vulnerability, or offer more source-agnostic evaluation, remains an open question with direct implications for human-AI collaboration. We examine this issue using logical fallacies as a controlled setting to isolate source-label effects on reasoning quality, independent of domain knowledge. We conduct an online study (N=505) where participants are assigned to a source condition (human, AI, human with AI assistance, AI with human assistance, or no disclosure) and evaluate comments containing logical fallacies, comparing their judgments with those of LLMs (GPT-5.2, Gemini 2.5 Flash, Claude Sonnet 4.5), who were evaluated across the same source conditions. Human evaluators were significantly more susceptible to fallacies labeled as written by human or human with AI assistance and assigned higher trust and evaluation ratings in these conditions. LLM evaluations remained comparatively stable across source labels, though performance varied across models. Confidence levels were similarly high across conditions for both humans and LLMs, regardless of fallacy presence. Our findings indicate that source-label bias in reasoning evaluation is primarily a human vulnerability and highlight the potential of human-LLM collaboration in increasingly AI-mediated environments.

Problem

• source bias
• logical fallacies
• human-AI collaboration
• reasoning evaluation
• label effects

Innovation

• source-label bias
• logical fallacies
• human-AI collaboration
• LLM evaluation
• reasoning judgment