🤖 AI Summary
The explosive growth of online video content has rendered manual brand safety review infeasible at scale. This paper presents the first systematic evaluation of multimodal large language models (MLLMs) for multilingual, multi-risk-category video brand safety assessment. We introduce the first fine-grained, high-accuracy multilingual video moderation dataset and benchmark leading MLLMs—including Gemini, GPT, and Llama—against domain-expert human reviewers. Results show that state-of-the-art MLLMs achieve human-level or near-human performance across most risk categories, substantially improving review efficiency and cost-effectiveness. Crucially, we identify prevalent failure modes, including misinterpretation of implicit semantics and cross-modal logical inconsistencies, providing empirically grounded insights for model refinement. The dataset is publicly released to foster reproducible research and industrial deployment in content safety.
📝 Abstract
As the volume of video content online grows exponentially, the demand for moderation of unsafe videos has surpassed human capabilities, posing both operational and mental health challenges. While recent studies demonstrated the merits of Multimodal Large Language Models (MLLMs) in various video understanding tasks, their application to multimodal content moderation, a domain that requires nuanced understanding of both visual and textual cues, remains relatively underexplored. In this work, we benchmark the capabilities of MLLMs in brand safety classification, a critical subset of content moderation for safe-guarding advertising integrity. To this end, we introduce a novel, multimodal and multilingual dataset, meticulously labeled by professional reviewers in a multitude of risk categories. Through a detailed comparative analysis, we demonstrate the effectiveness of MLLMs such as Gemini, GPT, and Llama in multimodal brand safety, and evaluate their accuracy and cost efficiency compared to professional human reviewers. Furthermore, we present an in-depth discussion shedding light on limitations of MLLMs and failure cases. We are releasing our dataset alongside this paper to facilitate future research on effective and responsible brand safety and content moderation.