VLM-CASE: Vision-Language Model Enabled Context-Adaptive Safety Envelopes for Anticipatory Safe Autonomous Driving

2026-07-06

AI Summary

This work addresses the challenge that adverse conditions such as inclement weather simultaneously degrade both perception and control capabilities in autonomous driving systems, a limitation exacerbated by the absence of human-like anticipatory mechanisms in existing approaches. The authors propose a context-adaptive safety framework that uniquely integrates a fine-tuned vision-language model (VLM) with formal safety theory. Specifically, a LoRA-fine-tuned VLM jointly infers road friction and visibility from front-view images, enabling dynamic generation of a safety envelope—termed CASE—based on Responsibility-Sensitive Safety principles and a friction budget. Within this envelope, any model-predictive controller can freely optimize trajectories. The framework is controller-agnostic, operates in real time, and demonstrates significant performance gains over conventional MPC and current VLM-integrated methods in closed-loop CARLA simulations, validating its effectiveness and generalizability across diverse environmental conditions.

Abstract

Adverse driving conditions, such as bad weather, remain a principal barrier to autonomous driving because they degrade two things at once: what the vehicle can perceive and what it can physically do. Human drivers cope by anticipation, reasoning about the scene and re-budgeting speed, following distance, and steering before grip or sight is lost, whereas current autonomous driving systems at best react after the fact. This paper proposes VLM-CASE, a framework that gives an autonomous vehicle this anticipatory capacity while keeping its motion bounded by a formal safety model at all times. A vision-language model (VLM), fine-tuned with low-rank adaptation (LoRA), reasons about the scene from the front-camera image and reports the road surface and visibility conditions. This output parametrizes a context-adaptive safety envelope (CASE), derived from physical limits and the guarantees of responsibility-sensitive safety, that couples braking and steering through a shared friction budget. A model predictive controller then drives freely within the envelope, while the VLM runs asynchronously so it never blocks the real-time control loop. We validate the framework in closed-loop CARLA simulation on tasks that demand both lateral and longitudinal control, across a range of weather, road-surface, and lighting conditions. The resulting controller, VLM-CASE-MPC, completes all trials, outperforming a conventional MPC baseline and a state-of-the-art VLM-integrated controller. Ablations confirm that the gains come from context adaptation, with the friction and visibility adaptations proving complementary. Furthermore, the framework is controller-agnostic and pairs with almost any low-level controller, offering a promising direction for safe autonomous driving. The dataset and supplementary materials for VLM-CASE are available at https://github.com/ytj254/VLM-CASE.

Problem

• autonomous driving
• adverse driving conditions
• anticipatory safety
• perception degradation
• vehicle control

Innovation

• Vision-Language Model
• Context-Adaptive Safety Envelope
• Anticipatory Driving
• Friction Budget
• Low-Rank Adaptation