Effectiveness of Zero-shot-CoT in Japanese Prompts

2025-03-09

AI Summary

Zero-shot chain-of-thought (Zero-shot-CoT) prompting is widely assumed to be universally effective across languages, yet its cross-lingual validity—particularly between Japanese and English—remains empirically unverified. Method: We conduct a systematic evaluation of Zero-shot-CoT on Japanese (JMMLU) and English (MMLU) benchmarks using GPT-3.5 and GPT-4o-mini, analyzing performance across diverse reasoning tasks. Contribution/Results: We uncover pronounced model dependency and task heterogeneity: Zero-shot-CoT yields substantial gains for Japanese tasks—especially university-level mathematics and abstract algebra—on GPT-3.5, but degrades overall performance on the more advanced GPT-4o-mini, with benefits persisting only in select high-difficulty Japanese reasoning tasks. These findings challenge the “plug-and-play” universality assumption of Zero-shot-CoT, demonstrating that its cross-lingual transfer efficacy is constrained by both model architecture and task-specific semantic complexity. Our study provides critical empirical evidence for designing multilingual reasoning prompts and underscores the need for language- and model-aware CoT strategies.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We compare the effectiveness of zero-shot Chain-of-Thought (CoT) prompting in Japanese and English using ChatGPT-3.5 and 4o-mini. The technique of zero-shot CoT, which involves appending a phrase such as"Let's think step by step"to a prompt to encourage reasoning before answering, has been shown to offer LLM performance improvements in mathematical and reasoning tasks, particularly in English. We investigate how these effects transfer to Japanese using the Japanese Multi-task Language Understanding Benchmark (JMMLU) and the Multi-task Language Understanding Benchmark (MMLU). Our results show that while zero-shot CoT prompting can lead to notable performance gains for some prompt categories in GPT-3.5, its impact in GPT-4o-mini is associated with significant performance declines. However, for Japanese prompts there remain certain categories, such as college mathematics and abstract algebra, that still exhibit improvements, despite the broader trend of diminishing effectiveness in more advanced models.

Problem

• Evaluates zero-shot CoT effectiveness in Japanese prompts.
• Compares performance gains between GPT-3.5 and GPT-4o-mini.
• Identifies categories like college math with improved results.

Innovation

• Zero-shot CoT prompting enhances reasoning tasks.
• Japanese prompts show varied effectiveness across models.
• Performance gains in specific Japanese academic categories.