Exploring Multimodal Challenges in Toxic Chinese Detection: Taxonomy, Benchmark, and Findings

2025-05-30

AI Summary

Large language models (LLMs) exhibit poor robustness against perturbations—such as character substitution—in Chinese multimodal toxic content detection. Method: We introduce the first Chinese multimodal toxicity benchmark dataset covering three perturbation categories (character substitution, visual noise, semantic deformation) and eight specific attack techniques; propose the first taxonomy for Chinese toxic multimodal perturbations; and systematically evaluate nine state-of-the-art Chinese and English LLMs under zero-shot, in-context learning (ICL), and supervised fine-tuning (SFT) paradigms. Contribution/Results: All SOTA models suffer >40% average accuracy drop on perturbed inputs; even minimal perturbation-based fine-tuning or prompting induces up to 35% false positives on benign samples—revealing a novel “over-correction” phenomenon in ICL/SFT. This work establishes a reproducible benchmark and a new robustness evaluation paradigm for Chinese multimodal content safety.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Detecting toxic content using language models is important but challenging. While large language models (LLMs) have demonstrated strong performance in understanding Chinese, recent studies show that simple character substitutions in toxic Chinese text can easily confuse the state-of-the-art (SOTA) LLMs. In this paper, we highlight the multimodal nature of Chinese language as a key challenge for deploying LLMs in toxic Chinese detection. First, we propose a taxonomy of 3 perturbation strategies and 8 specific approaches in toxic Chinese content. Then, we curate a dataset based on this taxonomy, and benchmark 9 SOTA LLMs (from both the US and China) to assess if they can detect perturbed toxic Chinese text. Additionally, we explore cost-effective enhancement solutions like in-context learning (ICL) and supervised fine-tuning (SFT). Our results reveal two important findings. (1) LLMs are less capable of detecting perturbed multimodal Chinese toxic contents. (2) ICL or SFT with a small number of perturbed examples may cause the LLMs"overcorrect'': misidentify many normal Chinese contents as toxic.

Problem

• Detecting perturbed toxic Chinese text challenges SOTA LLMs
• Multimodal Chinese language complicates toxic content detection
• ICL and SFT may cause overcorrection in LLM toxicity detection

Innovation

• Proposed taxonomy for toxic Chinese perturbations
• Benchmarked 9 SOTA LLMs on perturbed text
• Explored ICL and SFT for cost-effective enhancements