Large Language Models in Misinformation Ecosystems: Misuse, Defense, and Vulnerability

2026-07-11

AI Summary

The misuse of large language models (LLMs) has transformed disinformation into an ecosystem-level security threat encompassing content, social context, evidence sources, and verification processes. This work proposes the first unified role-based analytical framework to systematically characterize the multifaceted roles of LLMs within the disinformation ecosystem—as attackers, defenders, and vulnerable components—and integrates attack patterns, detection mechanisms, and defense strategies across a four-layer architecture. The study exposes critical limitations in current detection paradigms, synthesizes LLM-enabled offensive and defensive techniques, and identifies three key open challenges, notably the need to shift from static detection toward ecosystem-level risk assessment under budget constraints. These insights lay the theoretical groundwork and provide strategic direction for building auditable and robust defenses against AI-driven disinformation.

Abstract

Large language models (LLMs) have transformed misinformation from a primarily content-centric problem into a broader ecosystem-level security challenge. When misused, LLMs create risks beyond false content generation, enabling attacks on the social contexts, evidence sources, retrieval corpora, and verification workflows that misinformation defense depends on. In this paper, we introduce a role-layer framework to unify these risks and defenses. The role dimension characterizes LLMs as attackers, defenders, and vulnerable components of verification systems, while the layer dimension covers content, social contexts, evidence environments, and verification workflows. Guided by this framework, we organize LLM-enabled attacks, investigate LLM-based detection and verification methods, analyze vulnerabilities in LLM-centric detection paradigms, and discuss existing countermeasures against LLM-enabled attacks. Building on this synthesis, we identify three key open challenges: moving from static detection accuracy to budgeted ecosystem-level risk evaluation, hardening LLM-centered verification pipelines against adversarial manipulation, and deploying auditable human-in-the-loop verification systems for trustworthy real-world misinformation defense.

Problem

• misinformation
• large language models
• ecosystem-level security
• verification systems
• adversarial manipulation

Innovation

• role-layer framework
• LLM-enabled misinformation
• verification pipeline vulnerability
• ecosystem-level risk
• human-in-the-loop verification