Road Surface Condition Detection with Machine Learning using New York State Department of Transportation Camera Images and Weather Forecast Data

2025-10-07

AI Summary

New York State Department of Transportation (NYSDOT) relies heavily on manual inspections and visual interpretation of roadside camera footage for road condition assessment, resulting in low efficiency and delayed response—especially critical during winter. Method: This study constructs the first statewide, large-scale, manually annotated roadside image dataset covering six road surface states. We propose a multimodal random forest model that fuses CNN-extracted visual features with numerical weather forecast data. Crucially, we optimize for cross-camera generalization—training the model to perform robustly on unseen camera devices. Contribution/Results: Our model achieves 81.5% accuracy on completely unseen camera streams, enabling real-time, automated, statewide road condition monitoring. It significantly reduces manual labor costs and enhances the timeliness and scientific rigor of winter transportation operations and decision-making.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Model-Based Reasoning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

The New York State Department of Transportation (NYSDOT) has a network of roadside traffic cameras that are used by both the NYSDOT and the public to observe road conditions. The NYSDOT evaluates road conditions by driving on roads and observing live cameras, tasks which are labor-intensive but necessary for making critical operational decisions during winter weather events. However, machine learning models can provide additional support for the NYSDOT by automatically classifying current road conditions across the state. In this study, convolutional neural networks and random forests are trained on camera images and weather data to predict road surface conditions. Models are trained on a hand-labeled dataset of ~22,000 camera images, each classified by human labelers into one of six road surface conditions: severe snow, snow, wet, dry, poor visibility, or obstructed. Model generalizability is prioritized to meet the operational needs of the NYSDOT decision makers, and the weather-related road surface condition model in this study achieves an accuracy of 81.5% on completely unseen cameras.

Problem

• Automating road condition classification using camera images and weather data
• Reducing labor-intensive manual monitoring of winter road surfaces
• Improving operational decisions with machine learning models for transportation

Innovation

• Combining camera images with weather forecast data
• Using convolutional neural networks and random forests
• Achieving 81.5% accuracy on unseen cameras