Knowledge-to-Verification: Exploring RLVR for LLMs in Knowledge-Intensive Domains

2026-05-18

AI Summary

Existing reinforcement learning with verifiable rewards (RLVR) methods are limited in knowledge-intensive tasks due to their reliance on sparse, high-quality verifiable data and their focus on verifying only final answers while neglecting the reasoning process. This work proposes the Knowledge-to-Verification (K2V) framework, which effectively extends RLVR to knowledge-intensive domains for the first time by automatically synthesizing verifiable data and introducing a fine-grained chain-of-thought verification mechanism. This approach mitigates the challenges of sparse rewards and erroneous reasoning. Experimental results demonstrate that K2V substantially enhances the reasoning capabilities of large language models on knowledge-intensive tasks while preserving their general-purpose performance with minimal degradation.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has demonstrated promising potential to enhance the reasoning capabilities of large language models (LLMs) in domains such as mathematics and coding. However, its applications on knowledge-intensive domains have not been effectively explored due to the scarcity of high-quality verifiable data. Furthermore, current RLVR focuses solely on the correctness of final answers, leading to the limitations of flawed reasoning and sparse reward signals. In this work, we propose Knowledge-to-Verification (K2V), a framework that extends RLVR to knowledge-intensive domains through automated verifiable data synthesis, while enabling verification of the LLM's reasoning process. Extensive experiments demonstrate that K2V enhances the reasoning of LLM in knowledge-intensive domains without significantly compromising the model's general capabilities. This study also suggests that integrating automated data synthesis with reasoning verification is a promising direction to enhance model capabilities in these broader domains. Code is available at https://github.com/SeedScientist/K2V.

Problem

• knowledge-intensive domains
• verifiable rewards
• reasoning verification
• reinforcement learning
• large language models

Innovation

• Reinforcement Learning with Verifiable Rewards
• Knowledge-Intensive Reasoning
• Automated Data Synthesis
• Reasoning Verification
• Large Language Models