CRRL: A Causality-Based Reinforcement Learning Framework for Autonomous System Recovery

2026-07-03

AI Summary

This work addresses the limitations of conventional reinforcement learning in autonomous system recovery, which lacks causal understanding, struggles to generalize to unseen faults, and often becomes trapped in stagnation states without effectively coordinating with rule-based recovery mechanisms. To overcome these challenges, the paper introduces Causal Reinforcement Learning for Recovery (CRRL), a novel framework that integrates causal reasoning directly into policy training. CRRL enables agents to proactively identify potential stagnation states and collaboratively trigger rule-based safety interventions. Built upon the MAPE-K architecture, the framework combines sensor data, online causal model construction, and a hybrid PPO-based policy optimization, leveraging causal relationships extracted from driving logs to guide learning. Experimental results across three driving scenarios demonstrate that CRRL significantly improves cumulative reward, traveled distance, and speed; notably, in roundabout scenarios, 45% of test runs required no external intervention, underscoring its effective synergy between autonomous navigation and recovery.

Abstract

Traditional reinforcement learning (RL) for recovery in autonomous systems lacks causal understanding and generalizes poorly to novel failure scenarios. RL policies often stall in failure states, spending up to 70% of an episode immobilized. Rule-based recovery alone is inadequate, and adding heuristic recovery to a pretrained PPO policy worsens rewards because policies cannot coordinate well with unanticipated interventions. The issue is not missing recovery mechanisms but a lack of policies trained to collaborate with them. We introduce CRRL, a causal-guided RL framework that trains policies to work effectively with rule-based recovery. The recovery detects stalled states and assists the agent. Causal relations from driving logs shape the training signal, teaching the policy to anticipate stalls and adjust actions in recovery contexts. The framework follows MAPE-K, with sensor collection, causal model construction, and hybrid RL policy training corresponding to Monitor, Analyze, and Plan/Execute, respectively. We evaluate CRRL through a four-condition ablation study across three driving scenarios, with 20 episodes per condition. We find that causal training significantly improves reward, distance, and velocity. Moreover, 9 of 20 roundabout episodes required zero recovery intervention, confirming navigation competence. These results show that causal-guided training produces effective RL policies that cooperate with rule-based safety components.

Problem

• reinforcement learning
• autonomous system recovery
• causality
• failure scenarios
• policy coordination

Innovation

• Causal Reinforcement Learning
• Autonomous System Recovery
• Rule-based Recovery Integration
• Causal Modeling
• MAPE-K Framework