DynaMark: A Reinforcement Learning Framework for Dynamic Watermarking in Industrial Machine Tool Controllers

2025-08-29

AI Summary

In Industry 4.0, highly networked CNC controllers are vulnerable to replay attacks; however, existing dynamic watermarking methods rely on linear Gaussian assumptions and fixed statistics, rendering them ill-suited for time-varying, nonlinear, and proprietary industrial systems. This paper proposes a reinforcement learning–based adaptive dynamic watermarking framework: watermark design is formulated as a Markov decision process, where zero-mean Gaussian watermark covariance is optimized online via Bayesian belief updates—without requiring prior system knowledge—to jointly ensure control performance, energy efficiency, and detection confidence. Evaluated on a Siemens Sinumerik 828D digital twin and a physical stepper motor platform, the method reduces watermark energy by 70%, induces no trajectory deviation, achieves an average detection delay of only one sampling period, and demonstrates rapid alarm response with minimal control performance degradation in real-world experiments.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Intelligent Robots: State Estimation
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Industry 4.0's highly networked Machine Tool Controllers (MTCs) are prime targets for replay attacks that use outdated sensor data to manipulate actuators. Dynamic watermarking can reveal such tampering, but current schemes assume linear-Gaussian dynamics and use constant watermark statistics, making them vulnerable to the time-varying, partly proprietary behavior of MTCs. We close this gap with DynaMark, a reinforcement learning framework that models dynamic watermarking as a Markov decision process (MDP). It learns an adaptive policy online that dynamically adapts the covariance of a zero-mean Gaussian watermark using available measurements and detector feedback, without needing system knowledge. DynaMark maximizes a unique reward function balancing control performance, energy consumption, and detection confidence dynamically. We develop a Bayesian belief updating mechanism for real-time detection confidence in linear systems. This approach, independent of specific system assumptions, underpins the MDP for systems with linear dynamics. On a Siemens Sinumerik 828D controller digital twin, DynaMark achieves a reduction in watermark energy by 70% while preserving the nominal trajectory, compared to constant variance baselines. It also maintains an average detection delay equivalent to one sampling interval. A physical stepper-motor testbed validates these findings, rapidly triggering alarms with less control performance decline and exceeding existing benchmarks.

Problem

• Detecting replay attacks on industrial controllers using outdated sensor data
• Adapting dynamic watermarking to time-varying nonlinear machine tool behaviors
• Balancing detection confidence with control performance and energy consumption

Innovation

• Reinforcement learning framework for adaptive watermarking
• Online policy adapting covariance using feedback
• Balances control performance and detection confidence