SteelBench: Evaluating Vision-Language Models in Real-World Industrial Environments

2026-07-06

AI Summary

Existing video benchmarks struggle to evaluate the capability of vision-language models in recognizing worker behaviors and reasoning about safety rules under real-world industrial surveillance conditions—such as low illumination, occlusion, and long-range viewing. This work proposes SteelBench, the first multidimensional diagnostic benchmark tailored to authentic steel plant environments. Constructed from 149 hours of surveillance footage, it comprises 1,345 densely annotated video clips curated via temporal deduplication, category balancing, and visibility-aware sampling. The dataset encompasses actions, personal protective equipment (PPE) attributes, spatial context, and explicit safety rules, and introduces a novel annotation provenance auditing mechanism. Experiments reveal that even the best-performing model achieves only 42.6% accuracy on action recognition (versus 84.6% for humans), exhibits safety judgment error rates of 37–58%, and fails to pass more than two diagnostic tests. Moreover, unaudited model-generated labels can inflate reported accuracy by up to 17 percentage points for related models.

Abstract

Existing video benchmarks evaluate action recognition on consumer videos, egocentric recordings, or simulated industrial environments. They do not test vision-language models under the visual and procedural conditions of real industrial CCTV, where workers appear as distant figures amid dust, steam, low light, glare, occlusion, and overlapping activities. We introduce STEELBENCH, a diagnostic benchmark for industrial surveillance that jointly evaluates per-worker activity recognition, safety-rule reasoning, and annotation provenance. SteelBench contains 1,345 densely annotated clips, curated from 149 hours of operational plant footage and 10,024 candidate clips using temporal deduplication, class balancing, and visibility-aware stratified sampling. Each clip includes dense per-worker action labels, PPE attributes, spatial context, and safety-rule annotations. Because model-assisted annotation can shape the labels later used for model evaluation, SteelBench includes a provenance-aware audit protocol. The protocol measures label influence, evaluates sensitivity to ground-truth provenance, and reports a human reference from expert-reviewed labels. Applying this audit, we find that unaudited VLM-sourced ground truth can inflate same-family model accuracy by up to 17 percentage points. Across nine VLMs from four architectural families, the best model reaches only 42.6% action accuracy, compared with an 84.6% human benchmark. Performance also fragments across recognition, robustness, calibration, and safety reasoning. Even when models predict the correct action, 37-58% of cases still yield incorrect safety judgments, and no model passes more than 2 of 5 diagnostic checks. The dataset is publicly available on Hugging Face.

Problem

• vision-language models
• industrial surveillance
• action recognition
• safety-rule reasoning
• annotation provenance

Innovation

• vision-language models
• industrial surveillance benchmark
• safety-rule reasoning
• annotation provenance
• diagnostic evaluation