VISTA: A Vision and Intent-Aware Social Attention Framework for Multi-Agent Trajectory Prediction

2025-11-13

AI Summary

Existing methods struggle to simultaneously ensure long-horizon goal-directedness and fine-grained social interaction modeling in multi-agent trajectory prediction, leading to trajectory distortion. To address this, we propose a vision-intention collaborative social attention framework. Our approach introduces two novel components: social token attention and pairwise attention graphs—extending single-agent goal-conditioned prediction to multi-agent cooperative forecasting while enabling interpretable visualization of social influence. The architecture integrates a recurrent goal-conditioned Transformer, a cross-attention fusion module, and joint intention-motion sequential modeling. Evaluated on MADRAS and SDD benchmarks, our method achieves state-of-the-art performance: collision rate on MADRAS drops from 2.14% to 0.03%; zero collisions are achieved on SDD; and significant improvements are observed in ADE, FDE, and minFDE metrics.

Technology Category

• Computer Vision: Multi-modal Vision
• Multiagent Systems: Adversarial Agents
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multi-agent trajectory prediction is crucial for autonomous systems operating in dense, interactive environments. Existing methods often fail to jointly capture agents'long-term goals and their fine-grained social interactions, which leads to unrealistic multi-agent futures. We propose VISTA, a recursive goal-conditioned transformer for multi-agent trajectory forecasting. VISTA combines (i) a cross-attention fusion module that integrates long-horizon intent with past motion, (ii) a social-token attention mechanism for flexible interaction modeling across agents, and (iii) pairwise attention maps that make social influence patterns interpretable at inference time. Our model turns single-agent goal-conditioned prediction into a coherent multi-agent forecasting framework. Beyond standard displacement metrics, we evaluate trajectory collision rates as a measure of joint realism. On the high-density MADRAS benchmark and on SDD, VISTA achieves state-of-the-art accuracy and substantially fewer collisions. On MADRAS, it reduces the average collision rate of strong baselines from 2.14 to 0.03 percent, and on SDD it attains zero collisions while improving ADE, FDE, and minFDE. These results show that VISTA generates socially compliant, goal-aware, and interpretable trajectories, making it promising for safety-critical autonomous systems.

Problem

• Predicting multi-agent trajectories in dense interactive environments
• Capturing long-term goals and fine-grained social interactions jointly
• Reducing unrealistic multi-agent futures and trajectory collision rates

Innovation

• Recursive goal-conditioned transformer for multi-agent forecasting
• Cross-attention fusion module integrating intent and motion
• Social-token attention mechanism for flexible interaction modeling