Problem
Research questions and friction points this paper is trying to address.
Outside-the-Vehicle Referencing
ego-motion
referential ambiguity
Speak to the City: Multimodal Resolution for Outside-the-Vehicle References
🤖 AI Summary
为解决自动驾驶车辆中由于自我运动和参照模糊造成的外部物体查询难题，研究提出一种融合用户视线和自然语言的多模态框架，并通过VR管道同步360度视频与车辆GNSS数据来训练轻量级Transformer网络。
📝 Abstract
As autonomous vehicles and Extended Reality (XR) headsets enable novel in-car interactions, seamlessly querying physical landmarks, known as Outside-the-Vehicle Referencing (OVR), remains challenging due to ego-motion and referential ambiguity. We present a robust, multimodal OVR framework fusing user gaze and natural language to identify Points of Interest (POIs). To address the scarcity of dynamic vehicular data, we developed a VR-based pipeline synchronizing 360-degree transit videos with vehicle GNSS telemetry. Through a user study (N=46) mapping passenger head orientation into a 3D geospatial Digital Twin, we captured authentic gaze-speech behaviors. We subsequently trained a lightweight Transformer network, leveraging LLMs to dynamically align continuous spatial gaze vectors with discrete verbal context. Experimental results demonstrate high accuracy and low computational overhead, achieving an 83.33% Top-1 accuracy (87.72% Top-2) and an average inference time of 24.3 milliseconds. This real-time paradigm effectively resolves referential ambiguity, enabling context-aware spatial retrieval for passengers within the vehicle.
Innovation
Methods, ideas, or system contributions that make the work stand out.
multimodal framework
gaze and natural language
VR-based pipeline
lightweight Transformer network
context-aware spatial retrieval
🔎 Similar Papers
No similar papers found.
