FleetAgent: Teleoperation Assistant for Autonomous Fleets via Vectorized V2N Messages

2026-06-19

AI Summary

This work addresses the challenges of high upstream bandwidth costs and low operator efficiency when large-scale autonomous fleets rely on remote human operators to handle rare failure cases. The authors propose FleetAgent, a cloud-based multimodal large language model assistant that leverages VecFormer to map compact, vectorized vehicle-to-network (V2N) messages into embeddings. A differentiable Top-K context selection mechanism is introduced to drastically reduce context length and KV cache overhead while preserving performance. FleetAgent generates structured natural language responses along with intervention urgency scores to prioritize operator decision-making. Experiments demonstrate a 625× reduction in upstream data load compared to raw image transmission, a 16.54× decrease in KV cache usage relative to text-based descriptions, and on the newly curated VecEval dataset, a 16.8% improvement in Lingo-Judge scores and a 19.9% reduction in intervention failure rate.

Abstract

Large-scale autonomous fleets rely on teleoperation to resolve rare failures, yet streaming raw sensor data from many vehicles is costly, and remote operators can only monitor a limited number of vehicles at a time. We introduce FleetAgent, a cloud-hosted multimodal large language model (MLLM) assistant that consumes compact vectorized vehicle-to-network (V2N) messages, such as map elements, detected objects, and the ego planned path. It provides a structured natural-language response (including narration, explanation, and evaluation of the plan and scene), along with an intervention urgency score for operator prioritization. To make structured messages compatible with token-based MLLMs, we propose VecFormer, a vector-to-embedding interface with differentiable top-K context selection that bounds context length and GPU KV-cache growth, enabling more efficient batch processing, which is important under the context of cloud-hosted large-scale fleet management. We also construct VecEval, a nuScenes-derived dataset with paired human and synthetic imperfect plans and human-verified language labels, to facilitate the training and evaluation of our proposed system. Our proposed system can reduce uplink payload by up to 625 times compared with raw images and reduce KV-cache memory by 16.54 times compared with original text descriptions. On VecEval, FleetAgent improves Lingo-Judge score by 16.8% and reduces intervention failure rate by 19.9%, compared with Qwen2.5-VL-7B using language descriptions. These results demonstrate that FleetAgent can utilize compact structured V2N messaging to enable efficient, explainable teleoperation monitoring for autonomous fleets.

Problem

• autonomous fleets
• teleoperation
• sensor data streaming
• operator monitoring
• V2N messaging

Innovation

• Vectorized V2N Messages
• VecFormer
• Teleoperation Assistant
• Multimodal LLM
• KV-cache Efficiency