Distilling Collaborative Dynamics into Latent Space for Implicit Coordination in Decentralized Multi-Agent Manipulation

2026-06-22

AI Summary

This work addresses the limitations of existing centralized approaches in multi-agent cooperative manipulation, which suffer from poor scalability and lack effective implicit coordination mechanisms under partial observability. The authors propose CLS-DP, a novel framework operating within the Centralized Training with Decentralized Execution (CTDE) paradigm that, for the first time, distills global collaborative dynamics into a shared latent space. This enables each agent to infer a collaboration-aware latent variable solely from its local RGB observations and task instructions, which then guides the denoising process of a diffusion policy—achieving efficient implicit coordination without explicit communication. Notably, the method’s computational overhead remains constant regardless of team size. Evaluated on six tasks in RoboFactory, CLS-DP achieves an average success rate of 38%, substantially outperforming both the best centralized baseline (20%) and a decentralized ablation model without collaborative latent variables (9%), demonstrating superior performance and parameter efficiency.

Abstract

Multi-arm manipulation demands precise spatiotemporal coordination, yet many centralized approaches scale poorly as team size increases. To address this, we propose CLS-DP, a decentralized multi-agent framework that enables implicit coordination under partial observability without shared global views, explicit state information, or inter-agent communication. Under the centralized training and decentralized execution (CTDE) paradigm, CLS-DP distills privileged multi-agent dynamics into a latent space. At deployment, each agent infers a collaborative latent from its local RGB observation and a shared task instruction; it then conditions the diffusion denoising process on this latent. This design enables implicit coordination with a per-agent cost independent of team size. Across six RoboFactory benchmark tasks spanning two to four agents, CLS-DP achieves a 38% mean success rate, outperforming the best centralized baseline (20%) and a decentralized ablation without the collaborative latent (9%). It also maintains superior parameter efficiency across all agent configurations. Attribution maps show that an agent conditioned on the collaborative latent places high attribution on the joints and grippers of both itself and its teammates throughout execution. This suggests that the learned latent efficiently encodes collaborative dynamics from local observation, which facilitates implicit coordination in realistic settings characterized by partial observability.

Problem

• implicit coordination
• decentralized multi-agent manipulation
• partial observability
• collaborative dynamics
• latent space

Innovation

• latent space distillation
• implicit coordination
• decentralized multi-agent manipulation
• diffusion policy
• partial observability