Human-in-the-loop Reasoning For Traffic Sign Detection: Collaborative Approach Yolo With Video-llava

2024-10-07

arXiv.org

AI Summary

To address high false detection rates in traffic sign recognition (TSR) under adverse weather and low-quality data, as well as poor robustness in few-shot and occlusion scenarios, this paper proposes a human-in-the-loop inference framework. It dynamically injects real-time human verification signals into the YOLOv8 detection pipeline and integrates Video-LLaVA—a video-capable extension of Qwen-VL-2—to enable spatiotemporal semantic alignment and error attribution. Detection and visual-language understanding are jointly optimized via LoRA fine-tuning and attention-guided training. Evaluated on BDD100K-TS and TT100K-v2, the method achieves mAP@0.5 of 86.3% and 79.1%, reduces false detections by 37%, decreases human intervention frequency by 52%, and maintains end-to-end latency below 120 ms. This work introduces the first closed-loop “detection–understanding–feedback” mechanism for TSR, significantly enhancing accuracy and reliability in complex traffic environments—particularly for speed-limit sign recognition.

Technology Category

• Humans and AI: Human-in-the-loop Machine Learning
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Problem

• Improve YOLO's accuracy in detecting traffic signs
• Address detection failures in adverse weather conditions
• Enhance detection with human-in-the-loop reasoning and Video-LLava

Innovation

• Combines YOLO with Video-LLava for detection
• Uses human-in-the-loop reasoning for accuracy
• Enhances detection in adverse weather conditions