TraVEL: Trajectory-Guided Video Embedding Learning for Driving-Video Retrieval

2026-08-13

AI Summary

General-purpose multimodal embedding models struggle to distinguish fine-grained, action-centric driving events—such as left versus right turns or acceleration versus deceleration—due to their overreliance on static visual scenes and insufficient modeling of dynamic motion cues. To address this limitation, this work proposes TraVEL, a novel framework that first applies supervised fine-tuning of Qwen3-VL-Embedding on the nuReasoning dataset using InfoNCE loss, followed by trajectory-guided reinforcement learning. In the latter stage, ego-vehicle trajectory similarity serves as a reward signal, and embeddings are optimized via Group Relative Policy Optimization. TraVEL is the first approach to incorporate ego-trajectory as privileged supervision into single-vector embedding learning, significantly enhancing discrimination of both longitudinal and lateral motion semantics while preserving retrieval efficiency. Experiments show that TraVEL improves longitudinal mAP by 9.8 and 7.2 points and lateral mAP by 4.7 and 1.5 points over baselines on 2B and 8B models, respectively.

Abstract

Efficiently retrieving relevant clips from large-scale driving logs is essential for data curation, model development, and safety analysis. Structured and rule-based retrieval systems can explicitly target driving events, but typically require expert-defined rules, auxiliary data, and multi-stage perception pipelines. Multimodal embedding models offer a simpler and more efficient alternative by representing each video with a single searchable vector. However, general-purpose models often rely on shortcuts from static scene context and struggle to distinguish motion-centric events, such as turning left versus right or accelerating versus decelerating. In this work, we study how to adapt a general-purpose multimodal embedding model to driving-video retrieval. We first fine-tune Qwen3-VL-Embedding on paired clips and reasoning traces from nuReasoning using an InfoNCE objective. While this stage substantially improves overall retrieval, caption supervision alone remains insufficient for fine-grained motion understanding. We therefore introduce TraVEL (Trajectory-Guided Video Embedding Learning), a motion-aware fine-tuning framework that uses ego-trajectory similarity as a reward within Group Relative Policy Optimization. Trajectories serve only as privileged training supervision; retrieval still operates on single-vector video embeddings without ego poses, expert rules, or auxiliary perception outputs. We further construct a driving-video retrieval benchmark from nuReasoning. Experiments show that TraVEL improves motion-centric retrieval across model scales: relative to SFT, it raises longitudinal and lateral mAP by 9.8 and 4.7 points at 2B, with corresponding gains of 7.2 and 1.5 points at 8B. TraVEL thus combines physically grounded supervision with efficient embedding-based search.

Problem

• driving-video retrieval
• motion-centric events
• video embedding
• trajectory guidance
• fine-grained motion understanding

Innovation

• Trajectory-Guided Learning
• Video Embedding
• Driving-Video Retrieval
• Motion-Centric Understanding
• Privileged Supervision