Problem
Research questions and friction points this paper is trying to address.
Multimodal Large Language Models
Safety-Critical Settings
Fire-Smoke Understanding
Benchmark
Reliability
SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs
🤖 AI Summary
为了解决多模态大语言模型在火烟理解中的可靠性问题，研究通过创建包含83K张图片的SAFIRE基准并采用GPT-5.4辅助验证方法，提高了模型的安全关键推理能力。
📝 Abstract
Multimodal Large Language Models (MLLMs) show strong progress on vision-language tasks, yet their reliability in safety-critical settings remains underexplored. Fire-smoke understanding is central to public safety and disaster response, but most existing benchmarks lack diverse real-world scenarios and context-aware evaluation. We introduce SAFIRE, a large-scale benchmark for fire-smoke understanding in MLLMs, comprising 83K captioned images from 20 scenarios and 193K multiple-choice VQA (MCVQA) generated from a 9.7K-image subset, spanning 10 evaluation dimensions from basic perception to higher-order reasoning. A GPT-5.4-assisted multi-stage verification pipeline with MLLM majority voting ensures annotation quality. Evaluating ten open-source MLLMs (8B-38B) yields an average accuracy of 61.9%, exposing major gaps in safety-critical reasoning. We further show that adapting vision encoders with only 7% of our domain-specific data boosts fire-scene classification accuracy from 20.1% to 64.5%, indicating that carefully curated data can yield substantial gains even when data volume is limited. All datasets, models, and code are available at https://risys-lab.github.io/SAFIRE/.
Innovation
Methods, ideas, or system contributions that make the work stand out.
SAFIRE
Multimodal Large Language Models
Fire-Smoke Understanding
Safety-Critical Reasoning
Domain-Specific Data
🔎 Similar Papers
No similar papers found.
