Scaling Traffic Insights with AI and Language Model-Powered Camera Systems for Data-Driven Transportation Decision Making

2025-10-10

AI Summary

To address real-time traffic flow monitoring challenges posed by dynamic camera viewpoints and massive video streams, this paper proposes an end-to-end AI framework. First, a graph-structured viewpoint normalization method is designed to unify spatial representations across non-fixed surveillance cameras. Second, YOLOv11 is fine-tuned for high-accuracy multimodal object detection, and a domain-specific large language model (LLM) augmented with exemplar-enhanced prompting is introduced to enable efficient video stream parsing and automated summarization of traffic pattern evolution. Evaluated on 9 million images, the system supports high-resolution, long-duration, and large-scale traffic analysis. Empirical results from New York City reveal a 9% weekday passenger vehicle density reduction following congestion pricing implementation, a biphasic trend in freight traffic (initial decline followed by recovery), and sustained growth in pedestrian and cyclist activity. The framework provides a scalable, data-driven technical foundation for intelligent traffic management and evidence-based policy evaluation.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Accurate, scalable traffic monitoring is critical for real-time and long-term transportation management, particularly during disruptions such as natural disasters, large construction projects, or major policy changes like New York City's first-in-the-nation congestion pricing program. However, widespread sensor deployment remains limited due to high installation, maintenance, and data management costs. While traffic cameras offer a cost-effective alternative, existing video analytics struggle with dynamic camera viewpoints and massive data volumes from large camera networks. This study presents an end-to-end AI-based framework leveraging existing traffic camera infrastructure for high-resolution, longitudinal analysis at scale. A fine-tuned YOLOv11 model, trained on localized urban scenes, extracts multimodal traffic density and classification metrics in real time. To address inconsistencies from non-stationary pan-tilt-zoom cameras, we introduce a novel graph-based viewpoint normalization method. A domain-specific large language model was also integrated to process massive data from a 24/7 video stream to generate frequent, automated summaries of evolving traffic patterns, a task far exceeding manual capabilities. We validated the system using over 9 million images from roughly 1,000 traffic cameras during the early rollout of NYC congestion pricing in 2025. Results show a 9% decline in weekday passenger vehicle density within the Congestion Relief Zone, early truck volume reductions with signs of rebound, and consistent increases in pedestrian and cyclist activity at corridor and zonal scales. Experiments showed that example-based prompts improved LLM's numerical accuracy and reduced hallucinations. These findings demonstrate the framework's potential as a practical, infrastructure-ready solution for large-scale, policy-relevant traffic monitoring with minimal human intervention.

Problem

• Develops AI framework for scalable traffic monitoring using cameras
• Addresses dynamic camera viewpoints with graph-based normalization method
• Generates automated traffic summaries using domain-specific language models

Innovation

• Fine-tuned YOLOv11 model extracts traffic metrics
• Graph-based method normalizes dynamic camera viewpoints
• Domain-specific LLM generates automated traffic summaries