Metric Aggregation Divergence: A Hidden Validity Threat in Agent-Based Policy Optimization and a Contractual Remedy

2026-06-27

AI Summary

This study identifies and formally names the “Metric Aggregation Discrepancy” (MAD) problem, wherein inconsistent metric aggregation across pipeline stages in coupled agent-based modeling and multi-objective evolutionary algorithm systems leads to erroneous policy recommendations and irreproducible results. To address this, the authors propose a “metric contract” mechanism that enforces a unified metric extraction interface across the entire pipeline at the architectural level, ensuring computational consistency during scheduling. The approach incurs only approximately 3% runtime overhead while substantially improving reliability: in reproducing EpidemiOptim, it reduces policy recommendation error rates from 83% to near zero, and in the Lake Problem benchmark, it increases joint threshold success rates from 0.401 to 0.552.

Abstract

Metric aggregation divergence (MAD) is the silent inconsistency that arises when distinct pipeline stages in an agent-based model coupled with a multi-objective evolutionary algorithm (ABM+MOEA) independently re-implement how an outcome metric is extracted from simulation trajectories. Unlike deliberate analytical choices, MAD operates at the level of pipeline architecture: each stage is internally coherent, and the inconsistency becomes visible only when cross-stage outputs are compared. Code inspection of EpidemiOptim, a JAIR-published epidemic policy toolbox, reveals three structurally independent aggregation paths in peer-reviewed code. A faithful replication of this structure produces champion disagreement in 64.2% of independent runs (n=500, 95% CI: [59.9%, 68.3%]). In a 300-seed policy-flip experiment, divergent aggregation causes the optimizer to recommend the wrong champion in 83% of replications, with a mean welfare gap of 2.19 units and a Gini inequality gap of 0.050 units. In a follow-up inference audit, 3 of 249 flipped seeds cross the significance boundary itself. A complementary enterprise follow-up produces the predicted null under near-commensurable rankings (rho = 0.991), while a public upstream rerun of the Lake Problem DPS workflow shows that the archived published-path recommendation reaches joint-threshold success 0.401 whereas a shared contract-path rule reaches 0.552. We introduce the metric contract - a single shared callable enforced at dispatch time across all pipeline stages - as the remedy. Framed as standard engineering discipline applied to the cross-stage metric interface, the contract eliminates divergence by construction with approximately 3% runtime overhead.

Problem

• Metric Aggregation Divergence
• Agent-Based Modeling
• Multi-Objective Evolutionary Algorithm
• Pipeline Consistency
• Policy Optimization

Innovation

• Metric Aggregation Divergence
• metric contract
• agent-based modeling
• multi-objective evolutionary algorithm
• pipeline consistency