Distilling Vision-Language Models for Robust Traffic Sign Perception in Autonomous Vehicles

2026-08-09

AI Summary

This work addresses the limited robustness of existing traffic sign recognition models under physical adversarial attacks—such as shadows, natural illumination variations, and printed patches—and the common trade-off where defense mechanisms degrade clean accuracy. To overcome these challenges, the authors propose LAMDA, a novel framework that, for the first time, integrates a frozen OpenCLIP text encoder into this domain. LAMDA constructs a language-anchored dual-prototype bank and leverages an auxiliary alignment loss together with knowledge distillation to guide visual feature learning during training, without requiring adversarial examples or incurring additional inference overhead. Experiments demonstrate that LAMDA consistently enhances robustness across four backbone architectures and three types of physical attacks on both GTSRB and LISA datasets, achieving gains of up to 13.2 percentage points while preserving or even improving clean accuracy.

Abstract

Traffic sign recognition (TSR) models based on deep neural networks achieve strong clean-data performance but remain vulnerable to physically realizable adversarial attacks, including shadow perturbations, natural-light interference, and printed patches. Existing defenses often improve robustness against one attack type while degrading performance on others, and can reduce clean accuracy. We propose LAMDA (Language-Anchored Model for Direction Alignment), a training framework that transfers language-grounded structure into TSR models without using adversarial examples or adding inference-time overhead. LAMDA builds two fixed prototype banks from VLM-generated sign descriptions and class names using a frozen OpenCLIP text encoder, and uses them to supervise visual features through two complementary auxiliary losses during training. At inference, the adapter and prototype banks are discarded, leaving a standard backbone and classifier. Evaluated on GTSRB and LISA across four backbones and three physical attack types, LAMDA is the only method among ten evaluated that consistently improves robustness across all attack-backbone-dataset combinations, with gains of up to +12.5 pp under shadow attacks and +13.2 pp under natural-light attacks, while preserving or improving clean accuracy in nearly all cases.

Problem

• traffic sign recognition
• adversarial attacks
• robustness
• physical perturbations
• clean accuracy

Innovation

• vision-language models
• adversarial robustness
• traffic sign recognition
• prototype-based distillation
• language-guided training