CoStream: Composing Simple Behaviors for Generalizable Complex Manipulation

2026-06-24

AI Summary

This work addresses the challenge of performing complex manipulation tasks that simultaneously demand millimeter-level precision, sustained physical contact, and zero-shot generalization. The authors propose a composable behavioral framework that decouples manipulation capabilities into three reusable modules—semantic reasoning, prediction, and reactive control—which are integrated via right-multiplication under a unified SE(3) interface to generate real-time pose commands. This approach transcends conventional monolithic policies or rigid pipeline paradigms, enabling task generalization and disturbance recovery without retraining. By integrating foundation models, multimodal perception, keypoint trajectory prediction, and tactile feedback control, the system demonstrates strong performance across eight real-world tasks, excelling particularly in precision assembly and object transfer while exhibiting robustness to execution perturbations.

Abstract

Long-horizon, contact-rich complex manipulation tasks, such as seating a GPU into a PCIe slot, demand both millimeter high precision and out-of-the-box generalization to new tasks. Existing paradigms struggle to satisfy both: classical pipelines use brittle, task-specific interfaces to achieve high-precision control but require costly pipeline redesigns to adapt to new tasks, whereas monolithic end-to-end policies provide better generalization but lack high precision on complex, out-of-distribution tasks unless retrained with new data. Both paradigms share an implicit assumption: once a manipulation capability is acquired, it must be deployed as a rigid pipeline or monolithic whole, rather than being freely decomposed and recomposed. In this paper, we show that complex manipulation capabilities can emerge naturally from the composition of simple, independent behaviors. Rather than deploying a monolithic policy or a rigid pipeline, we propose \ourshort, a framework orchestrating foundation models and diverse sensing modalities into multiple composable core behaviors: a semantic behavior extracting spatial constraints via foundation models; a predictive behavior forecasting trajectories by tracking keypoints in imagined videos; and a reactive behavior providing high-frequency tactile and force corrections. On a shared $SE(3)$ interface, these outputs compose by right-multiplication into a single pose command at each control step, executed by a compliant controller. We demonstrate \ourshort on 8 real-world tasks spanning everyday manipulation and precision assembly, with the strongest gains in contact-rich assembly and object transfer, and show robust recovery from manual perturbations during execution. {Website:} https://costream-simple.github.io

Problem

• complex manipulation
• generalization
• high-precision control
• long-horizon tasks
• contact-rich tasks

Innovation

• composable behaviors
• foundation models
• SE(3) composition
• tactile feedback
• long-horizon manipulation