Closing the Loop: Motion Prediction Models beyond Open-Loop Benchmarks

2025-05-08

AI Summary

Motion prediction models often exhibit improved open-loop accuracy without corresponding gains in closed-loop driving performance, revealing a critical misalignment between standard evaluation metrics and real-world autonomy. Method: This paper introduces the first systematic prediction-planning co-evaluation framework for closed-loop assessment, integrating state-of-the-art learning-based predictors (e.g., Transformer- and GAN-based models) with realistic planners within end-to-end CARLA and nuScenes-based driving simulations. Contribution/Results: We empirically demonstrate that open-loop prediction accuracy is not a reliable proxy for closed-loop performance; instead, temporal consistency of predictions and compatibility with downstream planners are decisive factors. Notably, a lightweight model with 86% fewer parameters achieves superior closed-loop driving metrics. The work establishes a new paradigm for prediction-planning co-evaluation and releases open-source code to advance closed-loop–oriented motion prediction research.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Model-Based Reasoning
• Computer Vision: Motion & Tracking

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Fueled by motion prediction competitions and benchmarks, recent years have seen the emergence of increasingly large learning based prediction models, many with millions of parameters, focused on improving open-loop prediction accuracy by mere centimeters. However, these benchmarks fail to assess whether such improvements translate to better performance when integrated into an autonomous driving stack. In this work, we systematically evaluate the interplay between state-of-the-art motion predictors and motion planners. Our results show that higher open-loop accuracy does not always correlate with better closed-loop driving behavior and that other factors, such as temporal consistency of predictions and planner compatibility, also play a critical role. Furthermore, we investigate downsized variants of these models, and, surprisingly, find that in some cases models with up to 86% fewer parameters yield comparable or even superior closed-loop driving performance. Our code is available at https://github.com/continental/pred2plan.

Problem

• Evaluating motion prediction models in closed-loop autonomous driving scenarios
• Assessing if open-loop accuracy improvements enhance real-world driving performance
• Investigating downsized models for comparable closed-loop performance with fewer parameters

Innovation

• Evaluates motion predictors with motion planners
• Shows downsized models can match performance
• Highlights temporal consistency and planner compatibility