Truth or Sophistry? LoFa: A Benchmark for LLM Robustness Against Logical Fallacies

2026-06-29

AI Summary

Current large language models exhibit insufficient robustness against manipulative language such as logical fallacies and lack systematic evaluation of their resistance to persuasion. To address this gap, this work proposes LoFa, the first comprehensive benchmark specifically designed to assess resilience to logical fallacies. LoFa leverages multi-agent collaboration to generate paired data consisting of factual questions and corresponding fallacious arguments, and establishes a multi-turn adversarial debate framework to simulate sustained persuasive interference. We introduce a novel metric, LFR@k, which effectively distinguishes between model errors stemming from knowledge gaps versus sensitivity to fallacies. Experimental results reveal that mainstream large language models display markedly varying vulnerabilities across different fallacy types, providing critical empirical insights for enhancing reasoning reliability.

Abstract

Large Language Models (LLMs) exhibit strong semantic capabilities, yet their resilience to manipulative linguistic patterns such as logical fallacies remains underexplored. Prior work has primarily examined whether LLMs can identify or classify fallacies, leaving their robustness against fallacious persuasion insufficiently studied. To address this gap, we introduce LoFa (Logical Fallacy), a comprehensive benchmark for evaluating LLM robustness against fallacies. LoFa is constructed through a multi-agent pipeline that pairs factual questions with fallacious arguments, and is accompanied by a multi-round debate framework for assessing model resilience under sustained adversarial persuasion. To disentangle fallacy robustness from a model's inherent knowledge limitations, we further propose Logical Fallacy Resistance at k (LFR@k), a metric that quantifies resistance to fallacious attacks. Experiments show that LLMs exhibit varying levels of robustness across different fallacy types, revealing distinct vulnerability profiles among models.

Problem

• Logical Fallacies
• LLM Robustness
• Adversarial Persuasion
• Fallacy Resistance

Innovation

• Logical Fallacy
• LLM Robustness
• Adversarial Persuasion
• Multi-agent Benchmark
• LFR@k