Inside madupite: Technical Design and Performance

2025-07-30

AI Summary

This paper addresses the exact solution of large-scale discounted infinite-horizon Markov decision processes (MDPs), where the state and action spaces are finite but too large to fit in single-machine memory and the discount factor is close to one. We propose the first fully distributed framework for exact MDP solving. Methodologically, we formulate the problem via mathematical optimization and design scalable, distributed policy iteration and linear programming solvers that tightly integrate storage and computational capabilities of HPC clusters—without resorting to function approximation. Key contributions include: (i) overcoming memory bottlenecks to enable exact solutions for ultra-large MDPs under near-undiscounted settings; (ii) supporting user-defined structured acceleration strategies; and (iii) demonstrating substantial scalability and efficiency gains on real-world applications—including epidemiological modeling and classical control benchmarks—establishing our solver as the only exact method capable of efficiently handling memory-resident-infeasible MDPs.

Technology Category

• Multiagent Systems: Distributed Problem Solving
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Search and Optimization: Distributed Search

Application Category

• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

In this work, we introduce and benchmark madupite, a newly proposed high-performance solver designed for large-scale discounted infinite-horizon Markov decision processes with finite state and action spaces. After a brief overview of the class of mathematical optimization methods on which madupite relies, we provide details on implementation choices, technical design and deployment. We then demonstrate its scalability and efficiency by showcasing its performance on the solution of Markov decision processes arising from different application areas, including epidemiology and classical control. Madupite sets a new standard as, to the best of our knowledge, it is the only solver capable of efficiently computing exact solutions for large-scale Markov decision processes, even when these exceed the memory capacity of modern laptops and operate in near-undiscounted settings. This is possible as madupite can work in a fully distributed manner and therefore leverage the memory storage and computation capabilities of modern high-performance computing clusters. This key feature enables the solver to efficiently handle problems of medium to large size in an exact manner instead of necessarily resorting to function approximations. Moreover, madupite is unique in allowing users to customize the solution algorithm to better exploit the specific structure of their problem, significantly accelerating convergence especially in large-discount factor settings. Overall, madupite represents a significant advancement, offering unmatched scalability and flexibility in solving large-scale Markov decision processes.

Problem

• Solves large-scale discounted infinite-horizon Markov decision processes
• Enables exact solutions for memory-intensive problems via distributed computing
• Allows customizable algorithms to exploit problem-specific structures for faster convergence

Innovation

• Distributed computing for large-scale MDPs
• Customizable algorithm for problem-specific acceleration
• Exact solutions without function approximations