Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs

2026-05-07

AI Summary

This study investigates whether reinforcement learning (RL) can enhance large language models’ direct recall of parametric knowledge rather than merely improving reasoning capabilities. Under a zero-shot, single-hop, closed-book question-answering setting, the authors train models using only binary correctness rewards and employ fact-level deduplication to ensure that observed gains stem specifically from improved knowledge recall. The work provides the first evidence that RL can boost knowledge retrieval by reallocating probability mass over the model’s internal knowledge, with the most challenging samples—comprising approximately 18% of the training data—accounting for about 83% of the performance gain. Across three model families and multiple factual QA benchmarks, the approach achieves an average relative improvement of 27%, substantially outperforming various training and inference baselines.

Abstract

Reinforcement learning (RL) has achieved remarkable success in LLM reasoning, but whether it can also improve direct recall of parametric knowledge remains an open question. We study this question in a controlled zero-shot, one-hop, closed-book QA setting with no chain-of-thought, training only on binary correctness rewards and applying fact-level train-test deduplication to ensure gains reflect improved recall rather than reasoning or memorization. Across three model families and multiple factual QA benchmarks, RL yields ~27% average relative gains, surpassing both training- and inference-time baselines alike. Mechanistically, RL primarily redistributes probability mass over existing knowledge rather than acquiring new facts, moving correct answers from the low-probability tail into reliable greedy generations. Our data-attribution study reveals that the hardest examples are the most informative: those whose answers never appear in 128 pre-RL samples (only ~18% of training data) drive ~83% of the gain, since rare correct rollouts still emerge during training and get reinforced. Together, these findings broaden the role of RL beyond reasoning, repositioning it as a tool for unlocking rather than acquiring latent parametric knowledge.

Problem

• reinforcement learning
• parametric knowledge
• large language models
• factual recall
• zero-shot QA

Innovation

• Reinforcement Learning
• Parametric Knowledge
• Knowledge Recall
• Large Language Models
• Probability Redistribution