Accurate Open-Loop Control of a Soft Continuum Robot Through Visually Learned Latent Representations

2026-03-20

AI Summary

This study addresses the challenge of achieving high-precision open-loop control for soft continuum robots in the absence of visual feedback. The authors propose an interpretable, video-learned latent representation based on two-dimensional oscillators, integrating a Visual Oscillator Network (VON), an Attention-Broadcasting Decoder (ABCD), and Koopman operator theory to enable single-shot optimal open-loop control in latent space for tracking image-specified trajectories. This work presents the first demonstration that interpretable latent dynamics learned directly from videos can support long-horizon stable open-loop control of soft robots. The proposed ABCD mechanism substantially enhances control accuracy, and the combined VON–ABCD–Koopman model achieves the lowest mean squared error in simulation while successfully validating stable behaviors including static holding, extrapolated equilibrium, and relaxation dynamics.

Technology Category

• Intelligent Robots: Behavior Learning & Control
• Computer Vision: Video Understanding & Activity Analysis
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This work addresses open-loop control of a soft continuum robot (SCR) from video-learned latent dynamics. Visual Oscillator Networks (VONs) from previous work are used, that provide mechanistically interpretable 2D oscillator latents through an attention broadcast decoder (ABCD). Open-loop, single-shooting optimal control is performed in latent space to track image-specified waypoints without camera feedback. An interactive SCR live simulator enables design of static, dynamic, and extrapolated targets and maps them to model-specific latent waypoints. On a two-segment pneumatic SCR, Koopman, MLP, and oscillator dynamics, each with and without ABCD, are evaluated on setpoint and dynamic trajectories. ABCD-based models consistently reduce image-space tracking error. The VON and ABCD-based Koopman models attains the lowest MSEs. Using an ablation study, we demonstrate that several architecture choices and training settings contribute to the open-loop control performance. Simulation stress tests further confirm static holding, stable extrapolated equilibria, and plausible relaxation to the rest state. To the best of our knowledge, this is the first demonstration that interpretable, video-learned latent dynamics enable reliable long-horizon open-loop control of an SCR.

Problem

• soft continuum robot
• open-loop control
• latent dynamics
• visual learning
• trajectory tracking

Innovation

• soft continuum robot
• open-loop control
• latent dynamics
• attention broadcast decoder
• Visual Oscillator Networks