Compositional Motion Generation from Demonstration with Object-Centric Neural Fields

2026-07-08

AI Summary

This work addresses the challenge of data-efficient and generalizable robot motion generation, aiming to learn from few demonstrations and adapt across diverse scenarios. The proposed approach integrates object-centric neural fields with a temporal mixture-of-experts (MoE) architecture, decomposing complex behaviors into object-based motion primitives through spatiotemporal compositionality. It further introduces canonical neural fields and latent-conditioned deformations to model 3D scene structure. Innovatively combining vision-based structural priors with language instructions, the method achieves category-level, cross-scenario systematic generalization. Experiments demonstrate that it accomplishes long-horizon manipulation tasks in simulation using significantly less training data than baselines, while exhibiting robustness to noise and enabling language-driven 3D manipulation in real-world environments.

Abstract

Compositionality, by organizing complex behavior as combinations of simpler elements, enables robot learning that is scalable and data efficient. Leveraging this principle, we propose a generative learning-from-demonstration framework that enables compositional modeling of robotic behavior by connecting perception and motion through shared object-level representations. We render scenes from object-centric neural representations that integrate canonical neural fields with latent-conditioned deformations, capturing positional and geometric variations in a smooth, consistent, and interpretable way. For motion generation, a temporal mixture-of-experts (MoE) employs a gating mechanism to combine object-conditioned movement primitives over time, producing complete trajectories. This spatial-temporal compositionality maintains the data efficiency of movement primitives while grounding motion in visual structure, enabling systematic generalization across diverse scene configurations. In simulation, long-horizon manipulation tasks are successfully completed using the proposed model, which requires significantly less training data than other image-based baselines. Real-world experiments further demonstrate the method's robustness to noise, its ability to generalize at the category level through language-based segmentation models, and its capacity to operate directly on 3D scene representations.

Problem

• compositional motion generation
• learning from demonstration
• object-centric representation
• systematic generalization
• robotic manipulation

Innovation

• object-centric neural fields
• compositional motion generation
• movement primitives
• temporal mixture-of-experts
• learning from demonstration