MMBERT: Scaled Mixture-of-Experts Multimodal BERT for Robust Chinese Hate Speech Detection under Cloaking Perturbations

2025-08-01

AI Summary

In Chinese social media, hate speech frequently evades detection via multimodal obfuscation—such as lexical camouflage, pitch-altered speech, and metaphorical imagery—rendering unimodal text-based detection methods insufficiently robust. To address this, we propose MoE-BERT, a multimodal mixture-of-experts architecture tailored for Chinese hate speech detection. Our method integrates modality-specific experts with a shared self-attention mechanism and introduces a router-guided dynamic expert assignment strategy to stabilize joint training of MoE and BERT. Furthermore, we adopt a three-stage progressive training paradigm to enhance generalization under adversarial perturbations. Evaluated on multiple Chinese multimodal hate speech benchmarks, our approach consistently outperforms fine-tuned BERT baselines, state-of-the-art large language models, and in-context learning methods, achieving superior accuracy and significantly improved robustness in identifying covert hateful content.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Hate speech detection on Chinese social networks presents distinct challenges, particularly due to the widespread use of cloaking techniques designed to evade conventional text-based detection systems. Although large language models (LLMs) have recently improved hate speech detection capabilities, the majority of existing work has concentrated on English datasets, with limited attention given to multimodal strategies in the Chinese context. In this study, we propose MMBERT, a novel BERT-based multimodal framework that integrates textual, speech, and visual modalities through a Mixture-of-Experts (MoE) architecture. To address the instability associated with directly integrating MoE into BERT-based models, we develop a progressive three-stage training paradigm. MMBERT incorporates modality-specific experts, a shared self-attention mechanism, and a router-based expert allocation strategy to enhance robustness against adversarial perturbations. Empirical results in several Chinese hate speech datasets show that MMBERT significantly surpasses fine-tuned BERT-based encoder models, fine-tuned LLMs, and LLMs utilizing in-context learning approaches.

Problem

• Detect Chinese hate speech under cloaking perturbations
• Address lack of multimodal strategies in Chinese context
• Improve robustness with MoE-integrated BERT framework

Innovation

• Multimodal BERT with Mixture-of-Experts architecture
• Progressive three-stage training paradigm
• Router-based expert allocation for robustness