PseudoBench: Measuring How Agentic Auto-Research Fuels Pseudoscience

2026-06-16

AI Summary

This work addresses the vulnerability of large language model–driven autonomous research agents to sophisticated pseudoscientific narratives, which can mislead them into generating plausible yet erroneous research reports, thereby undermining scholarly credibility. To tackle this issue, the authors introduce PseudoBench, the first end-to-end benchmark designed to systematically evaluate agents’ capacity to detect and resist pseudoscience. PseudoBench encompasses 200 carefully constructed pseudoscientific claim–evidence pairs across five domains and spans the entire research pipeline—from experimental design to manuscript writing. Experiments on seven state-of-the-art agents reveal a pervasive lack of rejection mechanisms, with the highest resistance rate reaching only 27.4%, demonstrating that current systems are highly susceptible to propagating pseudoscience and underscoring both the urgency of this challenge and the necessity of the proposed benchmark.

Abstract

As Large Language Model based agents enter autonomous scientific research, their ability to resist pseudoscience becomes increasingly important. Otherwise, such systems may rapidly generate plausible yet misleading studies that contaminate academic literature and erode trust in science. We present PseudoBench, an adversarial benchmark for evaluating whether agentic auto-research systems can identify and resist pseudoscientific narratives. PseudoBench contains 200 curated pseudoscientific claim-evidence pairs across five domains and evaluates agents through an end-to-end research pipeline from experiments to writing. Testing seven state-of-the-art agents, we find that current systems readily produce persuasive reports that align with pseudoscientific premises with near-zero refusal rates and the highest resistance of only 27.4%. Stronger agents risk packaging pseudoscience in more sophisticated scientific language, increasing its apparent credibility. These findings reveal an alarming capacity to fuel pseudoscience, calling for scientific alignment before widespread deployment.

Problem

• pseudoscience
• agentic auto-research
• scientific alignment
• adversarial benchmark
• LLM agents

Innovation

• PseudoBench
• agentic auto-research
• pseudoscience detection
• adversarial benchmark
• scientific alignment