Unveiling Transferability in Trajectory Prediction via Latent Scene Embeddings

2026-06-29

AI Summary

This work addresses the limited generalization of trajectory prediction models in cross-dataset scenarios, primarily caused by discrepancies in scene layouts, agent behaviors, and perceptual conditions. The authors propose a transferability assessment framework based on latent scene embeddings and distributional distance metrics, establishing the first large-scale transfer experiment suite encompassing 24 mainstream trajectory datasets. Their analysis reveals a strong correlation between inter-dataset similarity and model transfer performance, enabling the design of a transferability scoring metric that effectively predicts cross-domain model behavior. This metric provides both theoretical grounding and practical guidance for pretraining strategies, dataset selection, and the development of foundational models for trajectory prediction.

Abstract

The growing availability of trajectory datasets has fueled major advances in data-driven motion prediction. Yet, models trained on one dataset often fail to generalize beyond their training domain as a result of differences in scene layouts, agent behaviors, and sensing conditions. A framework that learns latent representations of datasets and quantifies their similarity using distributional metrics is presented. This large-scale study covers 24 major datasets, including the most widely used motion-prediction benchmarks, and shows that the resulting transferability scores strongly correlate with cross-dataset model performance. The results provide practical guidance for dataset selection, pretraining, and large-scale foundation models for motion prediction, paving the way toward more generalizable and robust predictive systems.

Problem

• trajectory prediction
• transferability
• domain generalization
• dataset similarity
• motion prediction

Innovation

• latent scene embeddings
• transferability
• trajectory prediction
• distributional metrics
• foundation models