ORAN-DEFEND: Subspace Detection and Sanitization of Backdoor DRL xApps in Open RAN

2026-07-07

AI Summary

This work addresses the threat posed by third-party deep reinforcement learning (DRL) xApps that may embed backdoors capable of degrading Open RAN service quality upon exposure to stealthy trigger signals. To counter this, the authors propose a defense mechanism that requires no retraining: leveraging a small set of trusted trajectories, they construct a secure subspace via singular value decomposition and project observed key performance indicators (KPIs) onto this subspace to neutralize malicious behavior. Theoretical analysis establishes, for the first time, necessary and sufficient conditions under which linear projection enables exact recovery, revealing that the energy distribution of the trigger signal in the orthogonal complement of the secure subspace critically determines defense efficacy. The study further identifies an inherent limitation of linear detection when triggers co-locate with legitimate signals. Experiments on the Colosseum COLORAN dataset demonstrate 100% return recovery and ≥99.5% defense success against four distinct DRL backdoor attacks, validating both effectiveness and boundary conditions.

Abstract

Open Radio Access Networks (O-RAN) increasingly delegate near-real-time control to deep reinforcement learning (DRL) xApps obtained from third-party vendors, creating a new supply-chain attack surface. A backdoor policy behaves optimally until an adversary injects a covert trigger into the observed key performance indicator (KPI) telemetry, at which point it issues harmful control actions that degrade quality of service (QoS). We present ORAN-DEFEND, a retraining-free wrapper that sanitizes a frozen, potentially compromised xApp by projecting each KPI window onto a safe subspace estimated from a small number of trusted clean rollouts via singular value decomposition (SVD). We establish, both analytically and empirically, a precise recovery condition: the defense succeeds if the trigger energy concentrates in the orthogonal complement of the safe subspace, and we quantify this boundary through the trigger's $\Eperp$ energy fraction. On the Colosseum COLORAN dataset, we evaluate four structurally distinct DRL backdoor attacks, like TrojDRL, SleeperNets, BadRL, and Q-Incept, spanning inner-loop and outer-loop poisoning regimes and demonstrate $100\%$ return recovery and $\geq99.5\%$ defense success rate across all four when the subspace assumption holds. A geometry ablation reveals an intrinsic and previously uncharacterized limit of any linear projection defense: when the trigger collocates with the legitimate signal, the $\Eperp$ energy fraction governs recovery monotonically, and the linear residual detector collapses to chance even while a nonlinear classifier retains perfect separability.

Problem

• backdoor attack
• Open RAN
• deep reinforcement learning
• xApp security
• supply-chain threat

Innovation

• subspace projection
• backdoor defense
• deep reinforcement learning
• Open RAN
• singular value decomposition