The Pursuit of Diversity: Multi-Objective Testing of Deep Reinforcement Learning Agents

2025-10-16

AI Summary

Existing deep reinforcement learning (DRL) testing tools—such as INDAGO—optimize solely for the number of failing scenarios, limiting coverage of diverse scene configurations and failure modes. Method: This paper proposes INDAGO-Nexus, a multi-objective evolutionary framework for failure-scenario generation that jointly optimizes failure probability and scenario diversity. It incorporates behavioral feature distance and trajectory divergence as diversity metrics and employs Pareto-optimal front selection to overcome the limitations of single-objective optimization. Contribution/Results: Evaluated on autonomous driving and automatic parking tasks, INDAGO-Nexus discovers, on average, 83% more unique failures than INDAGO and accelerates fault detection by 67%. The approach significantly enhances the effectiveness and comprehensiveness of DRL agent testing in safety-critical scenarios.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Testing deep reinforcement learning (DRL) agents in safety-critical domains requires discovering diverse failure scenarios. Existing tools such as INDAGO rely on single-objective optimization focused solely on maximizing failure counts, but this does not ensure discovered scenarios are diverse or reveal distinct error types. We introduce INDAGO-Nexus, a multi-objective search approach that jointly optimizes for failure likelihood and test scenario diversity using multi-objective evolutionary algorithms with multiple diversity metrics and Pareto front selection strategies. We evaluated INDAGO-Nexus on three DRL agents: humanoid walker, self-driving car, and parking agent. On average, INDAGO-Nexus discovers up to 83% and 40% more unique failures (test effectiveness) than INDAGO in the SDC and Parking scenarios, respectively, while reducing time-to-failure by up to 67% across all agents.

Problem

• Discover diverse failure scenarios for DRL agents
• Overcome single-objective limitations in testing approaches
• Jointly optimize failure likelihood and test diversity

Innovation

• Multi-objective evolutionary algorithms optimize failure likelihood
• Multiple diversity metrics ensure varied test scenarios
• Pareto front selection balances failure diversity and probability