Multimodal Scenario Similarity Search for Autonomous Driving

2026-07-10

AI Summary

This work addresses the absence of efficient multimodal scene similarity retrieval methods in large-scale autonomous driving datasets that jointly account for visual appearance and dynamic behavior. The authors propose a unified multimodal retrieval framework that, for the first time, systematically integrates explicit trajectory matching (Exo-Trajectory) with Transformer-based trajectory representations (ScenarioFormer), alongside vision embeddings trained via contrastive learning. Experimental results demonstrate that trajectory-based representations excel in dynamic scenarios such as cut-ins and turns, while visual methods are better suited for appearance-dominated scenes. Crucially, multimodal fusion substantially enhances overall retrieval performance, revealing the complementary nature of appearance and motion features in assessing scene similarity.

Abstract

Large-scale autonomous-driving datasets contain vast numbers of recorded scenarios, creating a need for efficient retrieval methods that can identify situations similar to a given query. Existing approaches typically rely on either visual representations or motion-based descriptions, making it difficult to understand their relative strengths and limitations for scenario retrieval. In this work, we present a multimodal framework for autonomous-driving scenario retrieval that combines visual and trajectory-based representations within a unified retrieval pipeline. We investigate two trajectory-based approaches: Exo-Trajectory, an explicit matching method based on surrounding-agent motion, and ScenarioFormer, a transformer-based representation learned from object trajectories using contrastive learning. We compare these approaches against strong vision-based baselines and analyze their behavior across a diverse set of driving scenarios. Experimental results show that trajectory representations provide strong retrieval performance for motion-centric events such as cut-ins, turning maneuvers, and traffic queueing, while visual embeddings excel when appearance cues are informative. Most importantly, combining visual and trajectory information consistently improves retrieval quality, yielding the best overall performance. These findings demonstrate that appearance and motion capture are complementary notions of scenario similarity and motivate multimodal retrieval systems for autonomous-driving data mining, dataset curation, and scenario-based validation.

Problem

• autonomous driving
• scenario retrieval
• multimodal similarity
• trajectory representation
• visual embedding

Innovation

• multimodal retrieval
• trajectory representation
• visual embedding
• ScenarioFormer
• autonomous driving scenarios