MultiZebraLogic: A Multilingual Logical Reasoning Benchmark

2025-11-05

AI Summary

Existing logical reasoning benchmarks lack multilingual coverage and controllable difficulty. Method: We propose MultiZebraLogic—the first multilingual logical reasoning benchmark targeting the Germanic language family—employing 14 clue types and 8 distractor categories, integrated with rule-based automatic generation, multilingual text synthesis, tunable difficulty control, and red herring injection across nine Germanic languages. Contribution/Results: We construct a high-quality dataset comprising 128 (core) + 1,024 (extended) problems per language; enable cross-lingual fair evaluation and scalable topic/difficulty customization; and empirically demonstrate that 4×5 logic puzzles pose significant challenges to state-of-the-art LLMs, with distractor clues reducing average accuracy by 23.6%, thereby validating the benchmark’s sensitivity and effectiveness.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Measuring the full abilities of large language models (LLMs) requires benchmarks representing multiple tasks. We aim to create large, high-quality datasets for comparison of logical reasoning skills across several languages and of suitable difficulty for LLMs of various reasoning ability. We explore multiple ways of increasing difficulty. We generate zebra puzzles in multiple languages, themes, sizes and including 14 different clue types and 8 red herring types (uninformative clues). We find puzzle sizes 2x3 and 4x5 are sufficiently challenging for GPT-4o mini (a non-reasoning model) and o3-mini (a reasoning model), respectively. Including 5 red herrings decreases o3-mini puzzle-level accuracy on 4x5 puzzles by 15$pm$7 %. Scores of o3-mini on 4x5 puzzles are not significantly affected by use of English vs. Danish or the common houses theme vs. the country-specific smoerrebroed theme. We find no correlation between difficulty and the selected clue types. Datasets of 128+1024 puzzles are published as MultiZebraLogic in each of nine Germanic languages for sizes 2x3 and 4x5. We publish code for puzzle generation, designed for adaptablity into more languages and themes.

Problem

• Creating multilingual logical reasoning benchmarks for evaluating LLM abilities
• Developing zebra puzzles with varying difficulty across languages and clue types
• Assessing how puzzle complexity and red herrings impact model performance

Innovation

• Generates multilingual zebra puzzles with varied themes
• Incorporates red herrings and multiple clue types
• Provides adaptable code for puzzle generation