FALCON: Actively Decoupled Visuomotor Policies for Loco-Manipulation with Foundation-Model-Based Coordination

2025-12-03

AI Summary

Heterogeneous perceptual inputs from locomotion and manipulation tasks degrade policy performance in loco-manipulation. Method: This paper proposes a decoupled cross-modal coordination framework that separates locomotion and manipulation subsystems; leverages vision-language foundation models to establish a latent-space coordination mechanism; introduces an unlabeled phase-progress head for autonomous phase inference; incorporates a coordination-aware contrastive loss to model action compatibility; and integrates modular diffusion policies with a text-guided phase estimation network. Contributions/Results: Evaluated on diverse embodied manipulation tasks, the method significantly outperforms both centralized and decentralized baselines. It achieves superior cross-distribution generalization, enhanced robustness to sensory perturbations, and improved adaptability to complex, dynamic environments—demonstrating effective decoupling without sacrificing coordination fidelity.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Diffusion Models for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

We present FoundAtion-model-guided decoupled LoCO-maNipulation visuomotor policies (FALCON), a framework for loco-manipulation that combines modular diffusion policies with a vision-language foundation model as the coordinator. Our approach explicitly decouples locomotion and manipulation into two specialized visuomotor policies, allowing each subsystem to rely on its own observations. This mitigates the performance degradation that arise when a single policy is forced to fuse heterogeneous, potentially mismatched observations from locomotion and manipulation. Our key innovation lies in restoring coordination between these two independent policies through a vision-language foundation model, which encodes global observations and language instructions into a shared latent embedding conditioning both diffusion policies. On top of this backbone, we introduce a phase-progress head that uses textual descriptions of task stages to infer discrete phase and continuous progress estimates without manual phase labels. To further structure the latent space, we incorporate a coordination-aware contrastive loss that explicitly encodes cross-subsystem compatibility between arm and base actions. We evaluate FALCON on two challenging loco-manipulation tasks requiring navigation, precise end-effector placement, and tight base-arm coordination. Results show that it surpasses centralized and decentralized baselines while exhibiting improved robustness and generalization to out-of-distribution scenarios.

Problem

• Decouples locomotion and manipulation into specialized policies
• Restores coordination using vision-language foundation model
• Improves robustness in complex loco-manipulation tasks

Innovation

• Decouples locomotion and manipulation into specialized policies
• Uses vision-language foundation model for policy coordination
• Incorporates phase-progress head and contrastive loss for structure