MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models

2026-04-17

arXiv.org

AI Summary

This study addresses the lack of comprehensive safety benchmarks for omni-modal large language models by introducing the first multi-context safety evaluation framework encompassing audio, visual, and textual modalities. The benchmark comprises 1,196 scenarios and pioneers a minimal-pair mechanism distinguishing safe from unsafe inputs to isolate subtle risk factors. Through controlled experiments and chain-of-thought analysis, this work precisely evaluates model sensitivity to nuanced risks and cross-modal reasoning capabilities. Findings reveal significant deficiencies in current models regarding fine-grained risk identification and multimodal integration. These insights provide critical empirical evidence for advancing safety architectures and training strategies in omni-modal systems, establishing a foundational resource for future research on robust multimodal alignment and risk mitigation in increasingly capable AI systems.

Abstract

Existing multimodal safety benchmarks focus solely on visual inputs and cannot assess Omni Large Language Models (LLMs) that process vision, audio, and text. We introduce MCBench, a benchmark with 1196 scenarios spanning four safety categories that require integrating multiple modalities for accurate safety assessment. Each unsafe scenario is paired with a minimally different safe counterpart to assess model sensitivity. Our evaluations of state-of-the-art models reveal significant challenges. Omni LLMs struggle with subtle or non-physical risks but perform better when salient visual or acoustic cues are present. Analysis of reasoning traces shows that, although models can extract modality-specific information, they often fail to integrate these cues effectively for safety judgments. Our findings reveal that current Omni LLMs lack robust cross-modal reasoning in safety-critical settings, underscoring the need for improved architectures and training strategies for multimodal safety.

Problem

• Omni Large Language Models
• Multimodal Safety Assessment
• Cross-modal Reasoning
• Safety Benchmark

Innovation

• MCBench
• Omni Large Language Models
• Multimodal Safety Assessment
• Cross-modal Reasoning
• Minimal Pair Evaluation