Can Large Language Models Develop Strategic Reasoning? Post-training Insights from Learning Chess

2025-07-01

AI Summary

This work investigates whether large language models (LLMs) can acquire strategic reasoning capabilities in chess via reinforcement learning (RL). It identifies an inherent limitation: pretrained LLMs exhibit fundamental deficits in strategic understanding, severely constraining RL performance gains. To address this, the authors propose a knowledge distillation–based dense reward mechanism leveraging an action-value network—using high-accuracy chess engines to generate fine-grained, action-level feedback, integrated with supervised fine-tuning and RL policy optimization. Experiments demonstrate that dense rewards substantially outperform sparse rewards; however, all LLM variants remain markedly inferior to human experts, exposing a fundamental bottleneck of the pretraining paradigm in deep strategic modeling. The core contributions are (1) the first systematic empirical characterization of the RL plasticity boundary for strategic reasoning in LLMs, and (2) a transferable, distillation-augmented RL framework for strategic skill acquisition.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

While reinforcement learning (RL) for large language models (LLMs) has shown promise in mathematical reasoning, strategic reasoning for LLMs using RL remains largely unexplored. We investigate whether LLMs can develop strategic reasoning capabilities through RL in chess. To this end, we leverage a chess-pretrained action-value network to provide dense reward on the LLM's output move quality, which can be seen as a form of knowledge distillation. Our experiments show that our distillation-based dense rewards often outperform sparse binary rewards. However, surprisingly, all models plateau far below expert levels. We provide SFT and RL ablations on chess reasoning training and find evidence that this limitation stems from a deficit in the pretrained models' internal understanding of chess--a deficit which RL alone may not be able to fully overcome.

Problem

• Exploring strategic reasoning in LLMs via reinforcement learning in chess
• Comparing dense vs sparse rewards for improving LLM move quality
• Investigating pretrained model limitations in chess understanding

Innovation

• RL-based strategic reasoning for LLMs in chess
• Chess-pretrained action-value network for dense rewards
• Knowledge distillation outperforms sparse binary rewards