🤖 AI Summary
Contemporary representation learning methods struggle to effectively capture causal relationships in multi-agent interactions, particularly failing to model indirect effects mediated by intermediate agents. To address this, we propose a causal metric learning framework that (i) introduces causal annotations into metric learning for the first time, explicitly regularizing the latent space to enhance sensitivity to both direct and indirect causal effects; and (ii) designs a sim-to-real transfer paradigm that requires no ground-truth causal annotations in the real world, integrating cross-domain multi-task learning with counterfactual perturbation analysis to enable robust causal knowledge transfer. Evaluated on multiple pedestrian trajectory datasets, our method significantly improves out-of-distribution (OOD) generalization, increases causal awareness by 32%, and achieves over 87% of the upper-bound transfer performance—even without access to real-world causal labels.
📝 Abstract
Modeling spatial-temporal interactions among neighboring agents is at the heart of multi-agent problems such as motion forecasting and crowd navigation. Despite notable progress, it remains unclear to which extent modern representations can capture the causal relationships behind agent interactions. In this work, we take an in-depth look at the causal awareness of these representations, from computational formalism to real-world practice. First, we cast doubt on the notion of non-causal robustness studied in the recent CausalAgents benchmark. We show that recent representations are already partially resilient to perturbations of non-causal agents, and yet modeling indirect causal effects involving mediator agents remains challenging. To address this challenge, we introduce a metric learning approach that regularizes latent representations with causal annotations. Our controlled experiments show that this approach not only leads to higher degrees of causal awareness but also yields stronger out-of-distribution robustness. To further operationalize it in practice, we propose a sim-to-real causal transfer method via cross-domain multi-task learning. Experiments on pedestrian datasets show that our method can substantially boost generalization, even in the absence of real-world causal annotations. We hope our work provides a new perspective on the challenges and potential pathways towards causally-aware representations of multi-agent interactions. Our code is available at https://github.com/socialcausality.
