Bottom-Up Reputation Promotes Cooperation with Multi-Agent Reinforcement Learning

2025-02-04

AI Summary

In multi-agent systems, spontaneous emergence of consensus-based reputation judgments remains infeasible under private beliefs, hindering cooperative behavior. This paper proposes LR², a bottom-up reputation learning framework that requires no predefined norms or centralized modules. It employs a dual-strategy architecture—comprising a dilemma strategy for action selection and an evaluation strategy for reputation updating—enabling co-evolution of reputation formation and cooperation incentives. Leveraging local interactions, self-interested objective optimization, and distributed strategy adaptation, LR² significantly enhances cooperation rates and robustness in spatial social dilemmas. Crucially, it is the first to demonstrate that reputation-driven strategy clustering spontaneously constructs structured cooperative environments. The framework establishes a scalable, self-organizing paradigm for decentralized reputation generation and cooperative evolution in multi-agent systems.

Technology Category

• Multiagent Systems: Multiagent Learning
• Game Theory and Economic Paradigms: Adversarial Learning
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

Reputation serves as a powerful mechanism for promoting cooperation in multi-agent systems, as agents are more inclined to cooperate with those of good social standing. While existing multi-agent reinforcement learning methods typically rely on predefined social norms to assign reputations, the question of how a population reaches a consensus on judgement when agents hold private, independent views remains unresolved. In this paper, we propose a novel bottom-up reputation learning method, Learning with Reputation Reward (LR2), designed to promote cooperative behaviour through rewards shaping based on assigned reputation. Our agent architecture includes a dilemma policy that determines cooperation by considering the impact on neighbours, and an evaluation policy that assigns reputations to affect the actions of neighbours while optimizing self-objectives. It operates using local observations and interaction-based rewards, without relying on centralized modules or predefined norms. Our findings demonstrate the effectiveness and adaptability of LR2 across various spatial social dilemma scenarios. Interestingly, we find that LR2 stabilizes and enhances cooperation not only with reward reshaping from bottom-up reputation but also by fostering strategy clustering in structured populations, thereby creating environments conducive to sustained cooperation.

Problem

• Promotes cooperation in multi-agent systems.
• Addresses consensus on reputation without predefined norms.
• Enhances cooperation through bottom-up reputation learning.

Innovation

• bottom-up reputation learning
• local observations interaction
• strategy clustering enhancement