Vejde: A Framework for Inductive Deep Reinforcement Learning Based on Factor Graph Color Refinement

2025-09-11

AI Summary

This work addresses decision-making under structured state spaces—comprising object classes and relational dependencies—by proposing an inductive policy learning framework. Methodologically, it models MDP states as bipartite graphs, introduces factor graph coloring to guide message passing in graph neural networks, and jointly trains the policy network via supervised and reinforcement learning. The key contribution is the first integration of factor graph coloring into deep reinforcement learning, enabling cross-instance generalization of policies across varying problem scales and topologies. Empirical evaluation across 80 task instances from 8 RDDL domains demonstrates that the Veijde agent achieves average test performance on unseen instances comparable to a per-instance trained MLP baseline, and substantially outperforms the online planning algorithm Prost. These results validate its strong inductive transfer capability.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Graph-based Machine Learning
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

We present and evaluate Vejde; a framework which combines data abstraction, graph neural networks and reinforcement learning to produce inductive policy functions for decision problems with richly structured states, such as object classes and relations. MDP states are represented as data bases of facts about entities, and Vejde converts each state to a bipartite graph, which is mapped to latent states through neural message passing. The factored representation of both states and actions allows Vejde agents to handle problems of varying size and structure. We tested Vejde agents on eight problem domains defined in RDDL, with ten problem instances each, where policies were trained using both supervised and reinforcement learning. To test policy generalization, we separate problem instances in two sets, one for training and the other solely for testing. Test results on unseen instances for the Vejde agents were compared to MLP agents trained on each problem instance, as well as the online planning algorithm Prost. Our results show that Vejde policies in average generalize to the test instances without a significant loss in score. Additionally, the inductive agents received scores on unseen test instances that on average were close to the instance-specific MLP agents.

Problem

• Inductive policy functions for structured decision problems
• Generalizing reinforcement learning to varying problem sizes
• Handling richly structured states with object relations

Innovation

• Combines data abstraction with graph neural networks
• Uses neural message passing for latent states
• Handles varying problem sizes via factored representation