Multisensory Continual Learning: Adapting Pretrained Visuomotor Policies to Force

2026-06-29

AI Summary

This work addresses the challenge of efficiently transferring vision-only robotic policies to tasks involving additional sensory modalities—such as force feedback—without requiring re-pretraining, while preserving or even enhancing original performance. The authors propose MuSe, a novel framework that enables continual transfer from vision-only to force-augmented policies through multi-stage modality fusion, multisensory future state prediction, experience replay, and fine-tuning of the pretrained policy. Experimental results demonstrate that MuSe significantly improves performance on real-world, contact-rich manipulation tasks and maintains or boosts the agent’s proficiency in the original vision-based tasks.

Abstract

Robot manipulation often relies on sensory feedback beyond vision, particularly in contact-rich settings where force, tactile, or audio signals reveal interaction states that are not directly observable from images. However, these modalities are often hardware- and task-specific, and large-scale multisensory robot datasets remain scarce. As a result, it is impractical to pretrain policies with every sensor they may encounter. We study multisensory continual learning: adapting a pretrained robot policy to new tasks with newly introduced modalities while preserving performance under the original sensor suite. We propose MuSe, which incorporates limited multisensory data into pretrained vision-only policies through multi-stage fusion, multisensory future prediction, and experience replay over pretraining data. We instantiate MuSe by augmenting a pretrained vision-only policy with force-torque sensing and evaluate it on real-world manipulation tasks. Our experiments show that MuSe performs strongly on contact-rich finetuning tasks while preserving, and in some cases improving, performance on the original pretraining tasks. These results suggest that a modest multisensory dataset can improve general robot capabilities beyond the finetuning distribution. Project website: https://jadenvc.github.io/multisensory-continual-learning/

Problem

• multisensory learning
• continual learning
• robot manipulation
• sensor adaptation
• pretrained policies

Innovation

• multisensory continual learning
• visuomotor policy adaptation
• multi-stage fusion
• force-torque sensing
• experience replay