LAVQA: A Latency-Aware Visual Question Answering Framework for Shared Autonomy in Self-Driving Vehicles

2025-11-14

AI Summary

In high-uncertainty autonomous driving scenarios, communication latency and human operator response delays critically impair timely decision-making in shared control. To address this, we propose a latency-aware visual question answering (VQA) framework. Our method introduces the Latency-Induced Collision Map (LICOM)—the first formalism jointly modeling temporal delays and spatial uncertainty to visualize the dynamic evolution of vehicle safety regions. By integrating spatiotemporal risk visualization with dynamic collision mapping, the framework enables remote operators to perform high-level semantic decisions under latency constraints. The system unifies VQA-based reasoning, closed-loop control, and low-latency communication protocols. Evaluated in CARLA simulations, it reduces collision rates by over 8× compared to baseline approaches, significantly enhancing both safety and real-time responsiveness of shared autonomy systems.

Technology Category

• Computer Vision: Vision for Robotics & Autonomous Driving
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

When uncertainty is high, self-driving vehicles may halt for safety and benefit from the access to remote human operators who can provide high-level guidance. This paradigm, known as {shared autonomy}, enables autonomous vehicle and remote human operators to jointly formulate appropriate responses. To address critical decision timing with variable latency due to wireless network delays and human response time, we present LAVQA, a latency-aware shared autonomy framework that integrates Visual Question Answering (VQA) and spatiotemporal risk visualization. LAVQA augments visual queries with Latency-Induced COllision Map (LICOM), a dynamically evolving map that represents both temporal latency and spatial uncertainty. It enables remote operator to observe as the vehicle safety regions vary over time in the presence of dynamic obstacles and delayed responses. Closed-loop simulations in CARLA, the de-facto standard for autonomous vehicle simulator, suggest that that LAVQA can reduce collision rates by over 8x compared to latency-agnostic baselines.

Problem

• Addresses variable latency in shared autonomy for self-driving vehicles
• Integrates Visual Question Answering with dynamic risk visualization
• Reduces collision rates by accounting for network and human delays

Innovation

• Integrates Visual Question Answering with spatiotemporal risk visualization
• Uses dynamically evolving Latency-Induced Collision Map (LICOM)
• Reduces collision rates by over 8x in simulations