ChLogic: Evaluating Robustness of Logical Reasoning in Chinese Expressions

2026-06-16

AI Summary

This study addresses the robustness of large language models in Chinese logical reasoning and their performance disparity compared to English by introducing ChLogic, the first English–Chinese aligned benchmark for logical reasoning evaluation. Built upon formal logic templates, ChLogic comprises three subsets—generic, challenging, and Chinese-specific phenomena—each pairing an English premise with five Chinese surface variants to assess cross-lingual reasoning consistency. Through back-translation analysis and evaluations on mainstream models (Qwen3, Ministral, GLM), the work reveals a significant performance gap between English and Chinese. While back-translation improves results on generic tasks, its benefits diminish on challenging ones and even degrade performance for Qwen3-32B and GLM-5.1, highlighting the complex interplay among linguistic surface forms, translation artifacts, and model behavior.

Abstract

Large language models perform increasingly well on standardized logical reasoning benchmarks, but whether this ability remains robust beyond English is unclear. We introduce ChLogic, an English--Chinese aligned benchmark that tests whether models preserve logical reasoning performance when the same latent logical structure is expressed in English and diverse Chinese surface realizations. Built from formal logical templates, the benchmark contains three data sets: (i) the General aligned set, derived from 60 General Propositions across nine template families; (ii) the Difficult aligned set, derived from 40 Difficult Problems; and (iii) the Chinese-only set, covering 15 language-specific phenomenon types. Each aligned item pairs one English reference expression with five Chinese realizations. Experiments on Qwen3, Ministral, and GLM models reveal a persistent English--Chinese performance gap. Back-translation from standard Chinese into English often improves performance on the General aligned set, but produces mixed effects on the Difficult aligned set, where Qwen3-32B and GLM-5.1 perform worse after translation. These results indicate that Chinese surface realization, translation artifacts, and model-specific behavior jointly affect multilingual logical reasoning. Overall, ChLogic provides a useful stress test for the robustness of multilingual reasoning.

Problem

• logical reasoning
• multilingual robustness
• Chinese expressions
• language alignment
• surface realization

Innovation

• multilingual logical reasoning
• Chinese surface realization
• aligned benchmark
• back-translation analysis
• robustness evaluation