Distances for Markov chains from sample streams

2025-05-23

AI Summary

Existing bisimulation metric computation relies on a fully known transition model, rendering it inapplicable to realistic settings where only sample trajectories—without explicit dynamics—are available. To address this, we propose the first streaming-sample-oriented framework for estimating bisimulation metrics. Our method introduces a novel representation grounded in optimal transport and linear programming, and innovatively employs a stochastic dual optimization algorithm for efficient computation. Theoretically, we derive a rigorous upper bound on sample complexity. Empirically, we validate estimation accuracy and convergence speed across multiple synthetic environments and real-world trajectory datasets. This work bridges a critical gap by establishing both theoretical foundations and practical algorithms for sample-driven bisimulation metric learning—previously unexplored in the literature.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Bisimulation metrics are powerful tools for measuring similarities between stochastic processes, and specifically Markov chains. Recent advances have uncovered that bisimulation metrics are, in fact, optimal-transport distances, which has enabled the development of fast algorithms for computing such metrics with provable accuracy and runtime guarantees. However, these recent methods, as well as all previously known methods, assume full knowledge of the transition dynamics. This is often an impractical assumption in most real-world scenarios, where typically only sample trajectories are available. In this work, we propose a stochastic optimization method that addresses this limitation and estimates bisimulation metrics based on sample access, without requiring explicit transition models. Our approach is derived from a new linear programming (LP) formulation of bisimulation metrics, which we solve using a stochastic primal-dual optimization method. We provide theoretical guarantees on the sample complexity of the algorithm and validate its effectiveness through a series of empirical evaluations.

Problem

• Estimating bisimulation metrics without transition dynamics
• Developing stochastic optimization for sample-based access
• Providing theoretical guarantees on sample complexity

Innovation

• Stochastic optimization for bisimulation metrics
• Linear programming formulation without transition models
• Sample access based metric estimation