Entropy-Augmented Multi-Objective Policy Optimization in Multiagent Systems

2026-08-12

AI Summary

This work addresses the issue of premature convergence in multi-agent multi-objective optimization, which often arises from behavioral homogenization. To mitigate this, the study introduces a behavioral entropy maximization mechanism into multi-objective evolutionary algorithms for the first time. Specifically, within the NSGA-II framework, it integrates policy entropy rewards with multi-objective fitness evaluation to explicitly promote behavioral diversity while preserving Pareto optimality. This approach effectively alleviates behavioral collapse and substantially enhances exploration capability. Experimental results in the rover domain demonstrate that, compared to the NSGA-II baseline, the proposed method achieves up to a 48% improvement in hypervolume metric, along with significantly enhanced solution set quality and diversity.

Abstract

Autonomous agent teams deployed in settings such as marine and extraterrestrial outposts must coordinate actions to achieve optimal outcomes across multiple competing objectives. Multi-objective evolutionary algorithms such as NSGA-II optimize for diversity in the objective space, but neglect diversity in the behavior space, possibly leading to premature convergence and a collapse in behaviors that may differentiate policies in different external conditions. To address this, we introduce an entropy-augmented policy evaluation strategy that incorporates an entropy bonus into agent fitness scores, discouraging behavioral homogeneity across the evolving population. By augmenting policy evaluation with a behavior-space diversity signal while preserving the underlying Pareto optimization framework, our method is designed to encourage exploration of behaviorally distinct policies in multiagent domains. We evaluate our approach across rover-domain experiments with qualitatively distinct reward structures and observe hypervolume improvements of up to 48% relative to the NSGA-II baseline, suggesting that behavioral diversity is a promising and underexplored direction for improving multi-objective multiagent evolutionary optimization.

Problem

• multi-objective optimization
• behavioral diversity
• multiagent systems
• evolutionary algorithms
• premature convergence

Innovation

• entropy augmentation
• behavioral diversity
• multi-objective optimization
• multiagent systems
• evolutionary algorithms