CSSBench: Evaluating the Safety of Lightweight LLMs against Chinese-Specific Adversarial Patterns

2026-01-02

arXiv.org

AI Summary

This work addresses the critical gap in safety evaluation benchmarks, which are predominantly designed for English and fail to capture Chinese-specific adversarial patterns—such as homophonic substitutions, Pinyin-based inputs, and character decomposition—that pose unique security risks to lightweight large language models (LLMs). To this end, we propose CSSBench, the first safety evaluation benchmark tailored for Chinese-language lightweight LLMs. CSSBench encompasses six high-risk domains, employs a multi-task evaluation framework, and features a test set constructed from real-world Chinese malicious queries incorporating diverse Chinese-specific adversarial strategies. Experimental results demonstrate that these linguistically grounded perturbations significantly undermine model safety and robustness, and that CSSBench effectively exposes critical vulnerabilities in current defenses, thereby offering a reliable tool for the safety assessment and deployment of Chinese lightweight LLMs.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Large language models (LLMs) are increasingly deployed in cost-sensitive and on-device scenarios, and safety guardrails have advanced mainly in English. However, real-world Chinese malicious queries typically conceal intent via homophones, pinyin, symbol-based splitting, and other Chinese-specific patterns. These Chinese-specific adversarial patterns create the safety evaluation gap that is not well captured by existing benchmarks focused on English. This gap is particularly concerning for lightweight models, which may be more vulnerable to such specific adversarial perturbations. To bridge this gap, we introduce the Chinese-Specific Safety Benchmark (CSSBench) that emphasizes these adversarial patterns and evaluates the safety of lightweight LLMs in Chinese. Our benchmark covers six domains that are common in real Chinese scenarios, including illegal activities and compliance, privacy leakage, health and medical misinformation, fraud and hate, adult content, and public and political safety, and organizes queries into multiple task types. We evaluate a set of popular lightweight LLMs and measure over-refusal behavior to assess safety-induced performance degradation. Our results show that the Chinese-specific adversarial pattern is a critical challenge for lightweight LLMs. This benchmark offers a comprehensive evaluation of LLM safety in Chinese, assisting robust deployments in practice.

Problem

• Chinese-specific adversarial patterns
• lightweight LLMs
• safety evaluation
• adversarial robustness
• language model safety

Innovation

• Chinese-specific adversarial patterns
• lightweight LLMs
• safety benchmark
• over-refusal
• CSSBench