SegDiff: Segmented Trajectory Diffusion for Consistent and Adaptive Robot Manipulation

2026-07-12

AI Summary

This work addresses the limitations of existing imitation learning approaches, which suffer from error accumulation in continuous action prediction and struggle with multimodal action distributions, as well as keyframe-based methods that rely on external planners and lack real-time responsiveness. To overcome these challenges, the authors propose SegDiff—a closed-loop visuomotor policy that decomposes demonstrations into motion segments between key poses. SegDiff leverages a diffusion model combined with DDIM inversion to predict continuous trajectories from the current state to the next key pose, and introduces a dynamic temporal ensembling mechanism to handle inconsistencies in multimodal sampling and adapt to environmental dynamics. Experiments demonstrate that SegDiff significantly outperforms prior methods in both simulation and real-world settings, exhibiting strong long-horizon reasoning, real-time adaptability, and control stability.

Abstract

Imitation learning enables robots to acquire manipulation skills from demonstrations by mapping observations to actions. Existing approaches predict either short-horizon continuous action sequences or discrete keyposes. However, continuous prediction methods suffer from compounding errors due to short prediction horizons and struggle with multi-modal action distributions, whereas keypose-based methods necessitate an external planner, constraining real-time applicability. To address these challenges, we introduce SegDiff, a closed-loop visuomotor policy that integrates the strengths of both paradigms. SegDiff decomposes demonstrations into motion segments between keyposes and learns to predict the continuous trajectory from the current state to the next keypose, enabling long-horizon prediction with real-time refinement. Furthermore, we leverage the capability of diffusion models and DDIM inversion to propose a Dynamic Temporal Ensembling mechanism, which allows the policy to efficiently respond to dynamic environments and mitigate discontinuities caused by inconsistent multi-modal sampling. SegDiff demonstrates significant performance gains over existing approaches across various simulated and real-world scenarios, indicating its strong ability to reason over extended temporal dependencies while maintaining real-time adaptability and control stability.

Problem

• imitation learning
• compounding errors
• multi-modal action distributions
• real-time applicability
• robot manipulation

Innovation

• trajectory diffusion
• imitation learning
• keypose-based planning
• dynamic temporal ensembling
• visuomotor policy