Distributed Denial of Science: How Indirect Data Poisoning of AI Systems Can Industrialize Scientific Fraud

2026-07-12

AI Summary

This study uncovers a novel form of indirect data poisoning attack in which remote adversaries manipulate publicly available datasets and upload them to open repositories, thereby inducing autonomous scientific agents to propagate false conclusions at scale—without requiring trigger tokens, fabricated papers, or proxy access. Empirical evaluation demonstrates that this attack achieves a success rate of 49.56% against three state-of-the-art large language models (Claude Opus 4.7, GPT-5.5, and Gemini 3.1 Pro), while current detection mechanisms identify only 6.0% of such incidents. To counter this threat, the authors propose a defense framework integrating scientist role modeling with data provenance auditing. Experimental results show that this approach reduces the attack success rate to 0%, substantially outperforming defenses based solely on role modeling, which still exhibit a 16.67% failure rate, thus establishing a new paradigm for ensuring integrity in AI-driven scientific research.

Abstract

Scientific fraud is the instrument of doubt that malicious entities can use to establish controversy in science. Historically, it required the resources of a company: deep pockets, ghostwritten articles, and corrupt academics. Today, Artificial Intelligence (AI) is increasingly automating scientific research, so we ask: Can a remote adversary weaponize the honest use of AI in science to compromise scientific integrity? We envision and empirically evaluate a new attack, indirect data poisoning, in which an adversary corrupts an open dataset and uploads the poisoned variant to a public repository. Autonomous research agents may independently retrieve and process this data, turning honest scientists into the unpaid and unwitting distributors of fraud at scale. Across five socially-salient topics, from hiring discrimination to the safety of autonomous vehicles, three widely used frontier AI systems (Claude Code with Claude Opus 4.7, Codex with GPT-5.5, Gemini CLI with Gemini 3.1 Pro), and 450 ethically contained experimental runs, we find that poisoning succeeds in 49.56% of runs, while the rate of poisoning detection is only 6.0%. The attack requires no topic-specific trigger-words, agent access, indirect prompt injection, or fabricated papers, only the open data ecosystem and misleading metadata. To mitigate the attacks, we propose and evaluate two measures: a scientist persona and a data provenance audit with five checks (referencing papers, social markers, statistical anomalies, related datasets, poisoning caution). We find that the persona still leaves 16.67% of runs with a poisoned conclusion, but provenance auditing reduces attack success rate to zero. Our results suggest that indirect data poisoning may enable scientific fraud at unprecedented scale, but these attacks can be mitigated with suitable auditing by agents during data retrieval.

Problem

• scientific fraud
• indirect data poisoning
• AI systems
• data integrity
• automated research

Innovation

• indirect data poisoning
• scientific fraud
• autonomous research agents
• data provenance audit
• AI-driven science