Crosslingual Reasoning through Test-Time Scaling

2025-05-08

AI Summary

This work investigates the cross-lingual (especially low-resource languages) and cross-domain (e.g., STEM → cultural commonsense) generalization of English-centric large language models (LLMs) after chain-of-thought (CoT) fine-tuning. We propose test-time reasoning computation scaling and controllable language-specific CoT prompting, revealing for the first time that increasing test-time compute significantly improves multilingual mathematical reasoning. We further design a “quote-then-think” cross-lingual reasoning paradigm, integrating multilingual prompt engineering with fine-grained reasoning behavior analysis. Experiments show our method enables English-centric LLMs to outperform baseline models with twice the parameter count on multilingual math benchmarks; high-resource languages yield more efficient and accurate reasoning, yet cross-domain generalization remains limited. Our core contributions are: (1) uncovering the mechanistic benefits of test-time scaling for multilingual reasoning, and (2) establishing the first controllable, interpretable cross-lingual CoT framework.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Reasoning capabilities of large language models are primarily studied for English, even when pretrained models are multilingual. In this work, we investigate to what extent English reasoning finetuning with long chain-of-thoughts (CoTs) can generalize across languages. First, we find that scaling up inference compute for English-centric reasoning language models (RLMs) improves multilingual mathematical reasoning across many languages including low-resource languages, to an extent where they outperform models twice their size. Second, we reveal that while English-centric RLM's CoTs are naturally predominantly English, they consistently follow a quote-and-think pattern to reason about quoted non-English inputs. Third, we discover an effective strategy to control the language of long CoT reasoning, and we observe that models reason better and more efficiently in high-resource languages. Finally, we observe poor out-of-domain reasoning generalization, in particular from STEM to cultural commonsense knowledge, even for English. Overall, we demonstrate the potentials, study the mechanisms and outline the limitations of crosslingual generalization of English reasoning test-time scaling. We conclude that practitioners should let English-centric RLMs reason in high-resource languages, while further work is needed to improve reasoning in low-resource languages and out-of-domain contexts.

Problem

• Generalizing English reasoning finetuning to multilingual contexts
• Improving multilingual math reasoning via inference compute scaling
• Controlling language of reasoning for better crosslingual performance

Innovation

• Scaling inference compute improves multilingual reasoning
• Quote-and-think pattern for non-English inputs
• Control language of CoT for efficient reasoning