KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance

2026-04-14

AI Summary

This work addresses the challenge of sparse rewards in reinforcement learning for complex reasoning with large language models, where existing prompting strategies suffer from redundancy, inconsistency, and high training overhead. The authors propose KnowRL, a novel framework that formulates prompt design as a minimal sufficient guidance problem. KnowRL dynamically constructs compact, interaction-aware knowledge subsets for reinforcement learning by decomposing prompts into atomic knowledge units and applying Constrained Subset Search (CSS), thereby resolving the pruning-interaction paradox among knowledge units. Evaluated on eight reasoning benchmarks, KnowRL-Nemotron-1.5B achieves an average accuracy of 70.08%, outperforming baseline methods by 9.63 points; when augmented with curated knowledge units, performance further improves to 74.16%, establishing a new state-of-the-art at the 1.5B parameter scale.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Knowledge Representation and Reasoning: Knowledge Representation Languages
• Machine Learning: Reinforcement Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

RLVR improves reasoning in large language models, but its effectiveness is often limited by severe reward sparsity on hard problems. Recent hint-based RL methods mitigate sparsity by injecting partial solutions or abstract templates, yet they typically scale guidance by adding more tokens, which introduce redundancy, inconsistency, and extra training overhead. We propose \textbf{KnowRL} (Knowledge-Guided Reinforcement Learning), an RL training framework that treats hint design as a minimal-sufficient guidance problem. During RL training, KnowRL decomposes guidance into atomic knowledge points (KPs) and uses Constrained Subset Search (CSS) to construct compact, interaction-aware subsets for training. We further identify a pruning interaction paradox -- removing one KP may help while removing multiple such KPs can hurt -- and explicitly optimize for robust subset curation under this dependency structure. We train KnowRL-Nemotron-1.5B from OpenMath-Nemotron-1.5B. Across eight reasoning benchmarks at the 1.5B scale, KnowRL-Nemotron-1.5B consistently outperforms strong RL and hinting baselines. Without KP hints at inference, KnowRL-Nemotron-1.5B reaches 70.08 average accuracy, already surpassing Nemotron-1.5B by +9.63 points; with selected KPs, performance improves to 74.16, establishing a new state of the art at this scale. The model, curated training data, and code are publicly available at https://github.com/Hasuer/KnowRL.

Problem

• reward sparsity
• reasoning
• hint-based reinforcement learning
• large language models
• guidance redundancy

Innovation

• Knowledge-Guided Reinforcement Learning
• Minimal-Sufficient Guidance
• Atomic Knowledge Points
• Constrained Subset Search
• Pruning Interaction Paradox