When and Which Sensor to Observe? Timely Tracking of a Joint Markov Source

2026-06-29

AI Summary

This work addresses the remote tracking problem under heterogeneous sensors, where both sampling cost and communication delay coexist. The authors propose a belief-state-based dynamic request scheduling method that models the problem as a continuous-state partially observable Markov decision process (POMDP) by constructing a sufficient statistic combining the joint Markov source state and the age of incorrect information. For the first time, this approach jointly optimizes the average age of incorrect information and sampling cost. Innovatively integrating model predictive control with watermarking-based trust calibration (MPC-WTC) and reinforcement learning (RL-MPC), the method enables efficient decision-making while handling partial observability. Numerical experiments demonstrate that the proposed scheme significantly reduces the weighted total cost, confirming its superiority over existing approaches.

Abstract

We investigate the problem of remote estimation (at a monitor) of a discrete-time joint Markov process with individual components which can be observed with dedicated sensors. At a given time slot, the monitor has the option of staying idle or sending a pull request to one of the sensors to obtain a partial state value, while the sensors are assumed to have heterogeneous sampling costs. Our goal is to develop a monitor pull policy, i.e., determining when and towards which sensor to send a pull request, in order to minimize a weighted sum of average age of incorrect information (AoII), or in short age, and sampling costs. As the communication model, we assume an erasure channel with a fixed one-slot delay from each sensor to the monitor. In this setting, the monitor does not perfectly know either the state of the process or the age, at any given time. We first obtain a sufficient statistic, namely belief, representing the joint distribution of the age and the current state of the observed process, by using the history of all pull requests and observations. Then, we formulate the optimization problem as a continuous state-space Markov decision process (MDP), namely belief-MDP, for the solution of which we propose two model predictive control (MPC) methods, namely MPC without terminal costs (MPC-WTC), and reinforcement learning MPC (RL-MPC). The effectiveness of the proposed methods is validated by numerical examples.

Problem

• remote estimation
• age of incorrect information
• Markov process
• sensor scheduling
• sampling cost

Innovation

• belief-MDP
• Age of Incorrect Information
• Model Predictive Control
• Remote Estimation
• Reinforcement Learning MPC