Beyond the Bellman Fixed Point: Geometry and Fast Policy Identification in Value Iteration

2026-04-19

AI Summary

This work revisits discounted Q-value iteration (Q-VI) through the lens of switched systems theory, addressing the limitations of classical contraction-based analyses in capturing the geometric structure of its trajectory—particularly their coarseness in characterizing when the greedy policy actually attains optimality. The paper formally introduces, for the first time, the notion of an “empirically optimal solution set” and reveals a two-phase convergence behavior: Q-VI first identifies the optimal action set in finitely many steps, then slowly converges to the optimal Q-function. By leveraging joint spectral radius analysis, the authors prove that the distance to this solution set decays exponentially at a rate strictly faster than the classical γ-contraction bound, thereby uncovering an intrinsic fast policy identification mechanism inherent in Q-VI.

Technology Category

• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Satisfiability
• Multiagent Systems: Adversarial Agents

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Dynamic programming is one of the most fundamental methodologies for solving Markov decision problems. Among its many variants, Q-value iteration (Q-VI) is particularly important due to its conceptual simplicity and its classical contraction-based convergence guarantee. Despite the central role of this contraction property, it does not fully reveal the geometric structure of the Q-VI trajectory. In particular, when one is interested not only in the final limit $Q^*$ but also in when the induced greedy policy becomes effectively optimal, the standard contraction argument provides only a coarse characterization. To formalize this notion, we denote by $\mathcal X^*$ the set of $Q$-functions whose corresponding tie-broken greedy policies are optimal, referred to as the practically optimal solution set (POS). In this paper, we revisit discounted Q-VI through the lens of switching system theory and derive new geometric insights into its behavior. In particular, we show that although Q-VI does not reach $Q^*$ in finite time in general, it identifies the optimal action class in finite time. Furthermore, we prove that the distance from the iterate to a particular subset of $\mathcal X^*$ decays exponentially at a rate governed by the joint spectral radius (JSR) of a restricted switching family. This rate can be strictly faster than the standard $γ$ rate when the restricted JSR is strictly smaller than $γ$, while the convergence of the entire $Q$-function to $Q^*$ can still be dominated by the slower $γ$ mode, where $γ$ denotes the discount factor. These results reveal a two-stage geometric behavior of Q-VI: a fast convergence toward $\mathcal X_1$, followed by a slower convergence toward $Q^*$ in general.

Problem

• Q-value iteration
• optimal policy identification
• geometric convergence
• practically optimal solution set
• Markov decision processes

Innovation

• value iteration
• switching systems
• joint spectral radius
• policy identification
• geometric convergence