Diagnosing Non-Markovian Observations in Reinforcement Learning via Prediction-Based Violation Scoring

2026-03-28

AI Summary

In reinforcement learning, observations are often corrupted by noise, delay, or partial observability, violating the Markov assumption; however, there is a lack of effective diagnostic tools to distinguish such non-Markovian effects from other causes of performance degradation. This work proposes a model-free, two-stage predictive framework that requires neither an environment model nor causal graphs: first, a random forest captures nonlinear dynamics to extract residuals, and then ridge regression tests whether incorporating historical observations significantly reduces residual prediction error, thereby quantifying non-Markovianity. The method introduces, for the first time, a bounded [0,1] model-free non-Markovianity score and reveals an “absorption phenomenon”—an anomalous score drop in low-dimensional environments. Evaluated across six environments, three algorithms, and multiple levels of AR(1) noise, the score correlates significantly with reward degradation in 13 out of 16 settings and successfully guides architecture selection, fully recovering performance lost due to non-Markovian observations.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Machine Learning: Reinforcement Learning
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Reinforcement learning algorithms assume that observations satisfy the Markov property, yet real-world sensors frequently violate this assumption through correlated noise, latency, or partial observability. Standard performance metrics conflate Markov breakdowns with other sources of suboptimality, leaving practitioners without diagnostic tools for such violations. This paper introduces a prediction-based scoring method that quantifies non-Markovian structure in observation trajectories. A random forest first removes nonlinear Markov-compliant dynamics; ridge regression then tests whether historical observations reduce prediction error on the residuals beyond what the current observation provides. The resulting score is bounded in [0, 1] and requires no causal graph construction. Evaluation spans six environments (CartPole, Pendulum, Acrobot, HalfCheetah, Hopper, Walker2d), three algorithms (PPO, A2C, SAC), controlled AR(1) noise at six intensity levels, and 10 seeds per condition. In post-hoc detection, 7 of 16 environment-algorithm pairs, primarily high-dimensional locomotion tasks, show significant positive monotonicity between noise intensity and the violation score (Spearman rho up to 0.78, confirmed under repeated-measures analysis); under training-time noise, 13 of 16 pairs exhibit statistically significant reward degradation. An inversion phenomenon is documented in low-dimensional environments where the random forest absorbs the noise signal, causing the score to decrease as true violations grow, a failure mode analyzed in detail. A practical utility experiment demonstrates that the proposed score correctly identifies partial observability and guides architecture selection, fully recovering performance lost to non-Markovian observations. Source code to reproduce all results is provided at https://github.com/NAVEENMN/Markovianes.

Problem

• Non-Markovian observations
• Reinforcement Learning
• Markov property violation
• Partial observability
• Diagnostic scoring

Innovation

• Non-Markovian diagnosis
• Prediction-based scoring
• Reinforcement learning
• Partial observability
• Markov property violation