Poly-Autoregressive Prediction for Modeling Interactions

2025-02-12

AI Summary

This work addresses ego-agent behavior prediction in multi-agent interactions, aiming to jointly model the ego-agent’s historical states and other agents’ real-time states while integrating physical constraints and intrinsic motivations. To this end, we propose the Poly-Autoregressive (PAR) framework: it unifies multi-agent states into a temporal token sequence, departing from conventional unidirectional autoregressive modeling to enable cross-scenario generalization; it employs a lightweight Transformer backbone for end-to-end sequence modeling and joint state tokenization, eliminating complex preprocessing. Evaluated on three diverse prediction tasks—human social motion, autonomous driving trajectory forecasting, and hand-object interaction object pose estimation—PAR consistently outperforms standard autoregressive baselines. Results demonstrate its superior representational capacity, robustness to domain shifts, and strong generalization across heterogeneous interaction scenarios.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Multiagent Systems: Adversarial Agents
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We introduce a simple framework for predicting the behavior of an agent in multi-agent settings. In contrast to autoregressive (AR) tasks, such as language processing, our focus is on scenarios with multiple agents whose interactions are shaped by physical constraints and internal motivations. To this end, we propose Poly-Autoregressive (PAR) modeling, which forecasts an ego agent's future behavior by reasoning about the ego agent's state history and the past and current states of other interacting agents. At its core, PAR represents the behavior of all agents as a sequence of tokens, each representing an agent's state at a specific timestep. With minimal data pre-processing changes, we show that PAR can be applied to three different problems: human action forecasting in social situations, trajectory prediction for autonomous vehicles, and object pose forecasting during hand-object interaction. Using a small proof-of-concept transformer backbone, PAR outperforms AR across these three scenarios. The project website can be found at https://neerja.me/PAR/.

Problem

• Predicts multi-agent behavior interactions
• Forecasts ego agent's future states
• Applies to social, vehicle, object scenarios

Innovation

• Poly-Autoregressive modeling for multi-agent
• Minimal data pre-processing changes required
• Transformer backbone enhances prediction accuracy