Structure-R1: Dynamically Leveraging Structural Knowledge in LLM Reasoning through Reinforcement Learning

2025-10-16

AI Summary

Large language models (LLMs) suffer from low information density and contextual fragmentation in complex reasoning tasks due to overreliance on unstructured textual inputs. Method: This paper proposes a structured knowledge-enhanced reasoning framework that dynamically generates task-adaptive, variable-structure representations via reinforcement learning and introduces a self-reward mechanism to validate structural correctness and completeness. The approach integrates retrieval-augmented generation (RAG), structured knowledge modeling, and end-to-end optimization to improve contextual information utilization efficiency. Contribution/Results: Evaluated on seven knowledge-intensive benchmarks, the method achieves accuracy comparable to or exceeding that of larger models (e.g., 13B/34B) using only a 7B-parameter LLM. These results empirically demonstrate the effectiveness and generalizability of structured knowledge representation in enhancing LLM reasoning capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) have demonstrated remarkable advances in reasoning capabilities. However, their performance remains constrained by limited access to explicit and structured domain knowledge. Retrieval-Augmented Generation (RAG) addresses this by incorporating external information as context to augment reasoning. Nevertheless, traditional RAG systems typically operate over unstructured and fragmented text, resulting in low information density and suboptimal reasoning. To overcome these limitations, we propose extsc{Structure-R1}, a novel framework that transforms retrieved content into structured representations optimized for reasoning. Leveraging reinforcement learning, extsc{Structure-R1} learns a content representation policy that dynamically generates and adapts structural formats based on the demands of multi-step reasoning. Unlike prior methods that rely on fixed schemas, our approach adopts a generative paradigm capable of producing task-specific structures tailored to individual queries. To ensure the quality and reliability of these representations, we introduce a self-reward structural verification mechanism that checks whether the generated structures are both correct and self-contained. Extensive experiments on seven knowledge-intensive benchmarks show that extsc{Structure-R1} consistently achieves competitive performance with a 7B-scale backbone model and matches the performance of much larger models. Additionally, our theoretical analysis demonstrates how structured representations enhance reasoning by improving information density and contextual clarity. Our code and data are available at: https://github.com/jlwu002/sr1.

Problem

• Enhancing LLM reasoning with structured knowledge representations
• Dynamically generating structural formats via reinforcement learning
• Improving information density and reliability in multi-step reasoning

Innovation

• Reinforcement learning generates dynamic structural representations
• Generative paradigm creates task-specific structures per query
• Self-reward mechanism verifies structural correctness and completeness