Visual Reasoning at Urban Intersections: FineTuning GPT-4o for Traffic Conflict Detection

2025-02-27

AI Summary

This study addresses the challenges of traffic conflict identification and low interpretability at signal-free urban intersections. We propose, for the first time, a bird’s-eye-view (BEV) video understanding framework based on fine-tuned multimodal large language models (MLLMs), specifically GPT-4o. Methodologically, the framework integrates visual perception with logical reasoning via supervised fine-tuning to achieve end-to-end conflict detection, while generating semantically grounded causal attributions and executable driving recommendations. Our key contribution lies in overcoming the limitation of conventional computer vision models—namely, their lack of high-level semantic reasoning—by introducing a human-in-the-loop evaluation system to ensure output reliability. Experimental results demonstrate strong performance: 77.14% accuracy in conflict detection, 89.9% in explanation fidelity, and 92.3% in action recommendation accuracy. These findings validate the feasibility and practical utility of MLLMs for real-time, interpretable traffic management.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Traffic control in unsignalized urban intersections presents significant challenges due to the complexity, frequent conflicts, and blind spots. This study explores the capability of leveraging Multimodal Large Language Models (MLLMs), such as GPT-4o, to provide logical and visual reasoning by directly using birds-eye-view videos of four-legged intersections. In this proposed method, GPT-4o acts as intelligent system to detect conflicts and provide explanations and recommendations for the drivers. The fine-tuned model achieved an accuracy of 77.14%, while the manual evaluation of the true predicted values of the fine-tuned GPT-4o showed significant achievements of 89.9% accuracy for model-generated explanations and 92.3% for the recommended next actions. These results highlight the feasibility of using MLLMs for real-time traffic management using videos as inputs, offering scalable and actionable insights into intersections traffic management and operation. Code used in this study is available at https://github.com/sarimasri3/Traffic-Intersection-Conflict-Detection-using-images.git.

Problem

• Detecting traffic conflicts at unsignalized urban intersections.
• Fine-tuning GPT-4o for visual and logical reasoning using videos.
• Providing real-time traffic management insights and driver recommendations.

Innovation

• Fine-tuned GPT-4o for traffic conflict detection.
• Used birds-eye-view videos for visual reasoning.
• Achieved high accuracy in explanations and recommendations.