PALM: Enhanced Generalizability for Local Visuomotor Policies via Perception Alignment

2026-01-27

AI Summary

This work addresses the poor generalization of image-based behavioral cloning in out-of-distribution scenarios—such as workspace shifts, viewpoint changes, and cross-embodiment transfer—by proposing a modular policy architecture that decomposes manipulation policies into coarse-grained global components and fine-grained local strategies. By incorporating visual attention mechanisms at the local policy level and conditioning them on proprioceptive representations, the method achieves invariance in local action distributions, enabling robust handling of diverse domain shifts without requiring additional modalities, architectural modifications, or new training data. Experiments demonstrate that the approach incurs only 8% and 24% performance degradation in simulation and real-world out-of-distribution settings, respectively, substantially outperforming baseline methods, which suffer degradations of 45% and 77%.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Multi-modal Vision
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Generalizing beyond the training domain in image-based behavior cloning remains challenging. Existing methods address individual axes of generalization, workspace shifts, viewpoint changes, and cross-embodiment transfer, yet they are typically developed in isolation and often rely on complex pipelines. We introduce PALM (Perception Alignment for Local Manipulation), which leverages the invariance of local action distributions between out-of-distribution (OOD) and demonstrated domains to address these OOD shifts concurrently, without additional input modalities, model changes, or data collection. PALM modularizes the manipulation policy into coarse global components and a local policy for fine-grained actions. We reduce the discrepancy between in-domain and OOD inputs at the local policy level by enforcing local visual focus and consistent proprioceptive representation, allowing the policy to retrieve invariant local actions under OOD conditions. Experiments show that PALM limits OOD performance drops to 8% in simulation and 24% in the real world, compared to 45% and 77% for baselines.

Problem

• generalization
• out-of-distribution
• visuomotor policy
• behavior cloning
• domain shift

Innovation

• Perception Alignment
• Local Visuomotor Policy
• Out-of-Distribution Generalization
• Behavior Cloning
• Modular Policy