An MDP Model for Censoring in Harvesting Sensors: Optimal and Approximated Solutions

2015-01-14

IEEE Journal on Selected Areas in Communications

AI Summary

This paper addresses energy-efficient information transmission for energy-harvesting sensors, aiming to maximize cumulative message utility (i.e., importance) under finite energy constraints. Method: We formulate the problem as an infinite-horizon Markov decision process (MDP) and—under a realistic battery dynamics model—rigorously prove that the optimal policy is a state-dependent importance-threshold truncation policy, where transmission decisions depend dynamically on the current battery level. Building on this structural insight, we propose a low-complexity, fast-converging model-driven stochastic approximation algorithm and benchmark it against Q-learning. Results: Experiments in both single-hop and multi-hop networks demonstrate that our algorithm significantly reduces computational overhead and accelerates convergence while achieving utility performance close to the theoretical optimum.

Technology Category

• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

In this paper, we propose a novel censoring policy for energy-efficient transmissions in energy-harvesting sensors. The problem is formulated as an infinite-horizon Markov Decision Process (MDP). The objective to be optimized is the expected sum of the importance (utility) of all transmitted messages. Assuming that such importance can be evaluated at the transmitting node, we show that, under certain conditions on the battery model, the optimal censoring policy is a threshold function on the importance value. Specifically, messages are transmitted only if their importance is above a threshold whose value depends on the battery level. Exploiting this property, we propose a model-based stochastic scheme that approximates the optimal solution, with less computational complexity and faster convergence speed than a conventional Q-learning algorithm. Numerical experiments in single-hop and multi-hop networks confirm the analytical advantages of the proposed scheme.

Problem

• Energy Harvesting
• Sensor Networks
• Optimal Energy Allocation

Innovation

• Energy Harvesting Sensors
• Reinforcement Learning
• Dynamic Information Importance Threshold