LogicPuzzleRL: Cultivating Robust Mathematical Reasoning in LLMs via Reinforcement Learning

2025-06-05

AI Summary

Large language models (LLMs) exhibit limited generalization on structured reasoning tasks, as supervised fine-tuning often induces domain-specific heuristics rather than robust, generalizable reasoning strategies. Method: We propose a “learning-by-playing” reinforcement learning framework that trains LLMs on seven custom-designed logic puzzles—spanning constraint propagation, spatial consistency, and symbolic deduction—using binary correctness rewards to drive iterative hypothesis generation and refinement. The approach employs policy gradient optimization without external symbolic tools or supervision. Contribution/Results: Our method significantly improves LLM generalization on medium-difficulty mathematical benchmarks (e.g., MATH, AMC), notably enhancing algebraic manipulation, geometric inference, and combinatorial reasoning. Crucially, it achieves, for the first time, the emergence of transferable structured reasoning capabilities in purely neural models through unsupervised, interactive puzzle solving—demonstrating that systematic multi-step reasoning can be internalized end-to-end without architectural or external tooling constraints.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models (LLMs) excel at many supervised tasks but often struggle with structured reasoning in unfamiliar settings. This discrepancy suggests that standard fine-tuning pipelines may instill narrow, domain-specific heuristics rather than fostering general-purpose thinking strategies. In this work, we propose a"play to learn"framework that fine-tunes LLMs through reinforcement learning on a suite of seven custom logic puzzles, each designed to cultivate distinct reasoning skills such as constraint propagation, spatial consistency, and symbolic deduction. Using a reinforcement learning setup with verifiable rewards, models receive binary feedback based on puzzle correctness, encouraging iterative, hypothesis-driven problem solving. We demonstrate that this training approach significantly improves out-of-distribution performance on a range of mathematical benchmarks, especially for mid-difficulty problems that require multi-step reasoning. Analyses across problem categories and difficulty levels reveal that puzzle training promotes transferable reasoning routines, strengthening algebraic manipulation, geometric inference, and combinatorial logic, while offering limited gains on rote or highly specialized tasks. These findings show that reinforcement learning over logic puzzles reshapes the internal reasoning of LLMs, enabling more robust and compositional generalization without relying on task-specific symbolic tools.

Problem

• Enhancing LLMs' structured reasoning in unfamiliar settings
• Developing general-purpose thinking via reinforcement learning on logic puzzles
• Improving multi-step mathematical reasoning without task-specific tools

Innovation

• Reinforcement learning fine-tunes LLMs on logic puzzles
• Binary feedback encourages hypothesis-driven problem solving
• Puzzle training enhances transferable multi-step reasoning skills