Hail to the Thief: Exploring Attacks and Defenses in Decentralised GRPO

2025-11-12

AI Summary

This paper identifies a novel security threat in decentralized Group Relative Policy Optimization (GRPO): malicious nodes can inject arbitrary adversarial tokens—either contextually or contextually agnostic—to poison the post-training process of large language models (LLMs). Method: We formally model and empirically validate this attack vector, demonstrating near-100% success rates within just 50 optimization iterations on mathematical and programming tasks. To counter it, we propose a dual-mode defense: (i) string-level gradient exchange constraints for homogeneous models, and (ii) consensus-based malicious token filtering for heterogeneous models. Contribution/Results: Our framework achieves 100% attack mitigation and establishes the first verifiable, highly robust security foundation for decentralized alignment training—enabling provably secure, collaborative LLM optimization without centralized oversight.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Group Relative Policy Optimization (GRPO) has demonstrated great utilization in post-training of Large Language Models (LLMs). In GRPO, prompts are answered by the model and, through reinforcement learning, preferred completions are learnt. Owing to the small communication volume, GRPO is inherently suitable for decentralised training as the prompts can be concurrently answered by multiple nodes and then exchanged in the forms of strings. In this work, we present the first adversarial attack in decentralised GRPO. We demonstrate that malicious parties can poison such systems by injecting arbitrary malicious tokens in benign models in both out-of-context and in-context attacks. Using empirical examples of math and coding tasks, we show that adversarial attacks can easily poison the benign nodes, polluting their local LLM post-training, achieving attack success rates up to 100% in as few as 50 iterations. We propose two ways to defend against these attacks, depending on whether all users train the same model or different models. We show that these defenses can achieve stop rates of up to 100%, making the attack impossible.

Problem

• Identifies adversarial attacks in decentralized GRPO training
• Demonstrates malicious token injection in LLM post-training
• Proposes defenses against poisoning attacks in collaborative learning

Innovation

• Adversarial attacks poison decentralized GRPO training
• Defenses achieve 100% stop rates against attacks
• Methods vary for same or different user models