When Helpers Become Hazards: A Benchmark for Analyzing Multimodal LLM-Powered Safety in Daily Life

2026-01-07

arXiv.org

AI Summary

This work addresses the behavioral safety risks posed by multimodal large language models (MLLMs), which may generate misleading content in everyday scenarios. To this end, the authors propose SaLAD, a benchmark comprising 2,013 real-world image–text samples spanning ten categories of common hazardous situations. SaLAD emphasizes safety assessments that require fine-grained cross-modal reasoning—tasks where textual information alone is insufficient—and uniquely focuses on behavioral safety in daily life, balancing genuinely dangerous cases with overly sensitive ones. The benchmark introduces an evaluation protocol centered on explicit safety warnings. Experiments on 18 leading MLLMs reveal that even the best-performing model achieves only a 57.2% safety response rate, highlighting significant limitations in current safety alignment approaches when applied to realistic multimodal contexts.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Large-scale security measurements

Abstract

As Multimodal Large Language Models (MLLMs) become an indispensable assistant in human life, the unsafe content generated by MLLMs poses a danger to human behavior, perpetually overhanging human society like a sword of Damocles. To investigate and evaluate the safety impact of MLLMs responses on human behavior in daily life, we introduce SaLAD, a multimodal safety benchmark which contains 2,013 real-world image-text samples across 10 common categories, with a balanced design covering both unsafe scenarios and cases of oversensitivity. It emphasizes realistic risk exposure, authentic visual inputs, and fine-grained cross-modal reasoning, ensuring that safety risks cannot be inferred from text alone. We further propose a safety-warning-based evaluation framework that encourages models to provide clear and informative safety warnings, rather than generic refusals. Results on 18 MLLMs demonstrate that the top-performing models achieve a safe response rate of only 57.2% on unsafe queries. Moreover, even popular safety alignment methods limit effectiveness of the models in our scenario, revealing the vulnerabilities of current MLLMs in identifying dangerous behaviors in daily life. Our dataset is available at https://github.com/xinyuelou/SaLAD.

Problem

• Multimodal LLM safety
• unsafe content
• daily life
• human behavior
• safety benchmark

Innovation

• Multimodal LLM Safety
• Safety Benchmark
• Cross-modal Reasoning
• Safety Warning Evaluation
• Real-world Risk Exposure