SPARK: Self-Play with Asymmetric Reward from Knowledge Graphs

2026-05-06

AI Summary

This work addresses the challenge of automatically generating trustworthy multi-hop reasoning questions from scientific literature, where relationships among multimodal elements are often implicit and difficult to verify. To this end, it introduces knowledge graphs into a self-play framework for scientific documents, constructing a unified graph to generate multi-hop relational questions and providing verifiable reward signals grounded in structured factual knowledge. By leveraging an information asymmetry mechanism, a single small-scale vision-language model alternately assumes the roles of questioner and answerer during training. The proposed approach significantly outperforms text-only self-play baselines on both public benchmarks and a newly curated cross-document multi-hop question answering dataset, with performance gains becoming more pronounced as the number of reasoning hops increases.

Abstract

Self-play reinforcement learning has shown strong performance in domains with formally verifiable structure, such as mathematics and coding, where both problem generation and reward computation can be grounded in explicit rules. Extending this paradigm to scientific literature is more challenging: the relationships among multi-modal elements within and across documents are rarely made explicit in text, which makes automatic generation of relational reasoning questions difficult and weakens the reliability of reward signals. We propose SPARK (Self-Play with Asymmetric Reward from Knowledge Graphs), a framework that automatically constructs a unified knowledge graph (KG) from multi-document scientific literature and uses it as the structural basis for self-play. KG paths over multimodal nodes serve as a source for generating relational reasoning questions, and structured facts stored in the KG provide a basis for verifiable reward computation. A single small vision-language model (sVLM) alternates between Proposer and Solver roles under information asymmetry against a fixed KG, a design that we believe can be naturally extended toward online adaptation in future work. We evaluate SPARK on public benchmarks and a self-constructed cross-document multi-hop QA dataset. Results show that SPARK consistently outperforms flat-corpus-based self-play baselines, and the performance gap widens as hop count increases, suggesting that KG-structure grounding contributes to relational multi-hop reasoning beyond what unstructured corpus grounding can provide.

Problem

• self-play reinforcement learning
• scientific literature
• relational reasoning
• knowledge graphs
• multi-hop QA

Innovation

• Self-play reinforcement learning
• Knowledge Graph
• Asymmetric Reward
• Multi-hop Reasoning
• Vision-Language Model