OnDeFog: Online Decision Transformer under Frame Dropping

2026-06-17

AI Summary

This work addresses the challenge of state frame loss in real-world reinforcement learning caused by communication delays or sensor failures. The authors propose a novel approach that integrates the DeFog frame-loss handling mechanism with an Online Decision Transformer (ODT), embedding state reconstruction and reward compensation directly into an end-to-end online learning framework for the first time. This integration enables robust policy optimization even under high frame loss rates. Compared to the original ODT and offline DeFog, the proposed method achieves significantly improved performance in high-loss environments and demonstrates superior generalization and adaptability in scenarios containing abundant low-return data.

Abstract

In challenging real-world reinforcement learning applications, communication delays or sensor failures often cause frame dropping, in which the agent cannot receive the dropped states and associated rewards. To address the performance degradation caused by frame dropping, the Decision Transformer under Random Frame Dropping (DeFog) was developed by incorporating additional mechanisms into the decision transformer to tackle frame dropping. Although DeFog can mitigate performance degradation in frame-dropping environments, since DeFog is an offline learning method, it struggles to effectively generalize to novel states not adequately represented in the training dataset. In this study, we propose OnDeFog, which integrates the mechanisms in DeFog with the online decision transformer (ODT), an online reinforcement learning method that learns policies through direct environmental interaction. Comprehensive experimental evaluation demonstrates that our proposed OnDeFog achieves superior performance compared to ODT in environments characterized by high dropping frame rate and outperforms DeFog on datasets containing a large amount of low-reward data.

Problem

• frame dropping
• reinforcement learning
• online learning
• offline learning
• generalization

Innovation

• Online Reinforcement Learning
• Decision Transformer
• Frame Dropping
• OnDeFog
• Robust Policy Learning