Beyond Egocentric Limits: Multi-View Depth-Based Learning for Robust Quadrupedal Locomotion

2025-11-27

AI Summary

Quadrupedal robots suffer degraded locomotion robustness under egocentric perception limitations—e.g., occluded fields of view. Method: We propose a multi-view depth perception–enhanced locomotion control framework that fuses depth streams from the robot’s primary camera and external heterogeneous cameras. To model cross-view knowledge transfer, we introduce a teacher–student distillation mechanism. Additionally, we employ domain randomization via stochastic remote-camera failure and 3D positional perturbations to improve perceptual robustness and Sim2Real generalization. Results: Experiments demonstrate significant performance gains over single-view baselines in dynamic tasks—including gap crossing and stair descent—while maintaining stable locomotion even under partial or complete external camera failure. The framework thus exhibits strong adaptability to complex environments and perceptual uncertainty.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Motion & Tracking
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recent progress in legged locomotion has allowed highly dynamic and parkour-like behaviors for robots, similar to their biological counterparts. Yet, these methods mostly rely on egocentric (first-person) perception, limiting their performance, especially when the viewpoint of the robot is occluded. A promising solution would be to enhance the robot's environmental awareness by using complementary viewpoints, such as multiple actors exchanging perceptual information. Inspired by this idea, this work proposes a multi-view depth-based locomotion framework that combines egocentric and exocentric observations to provide richer environmental context during agile locomotion. Using a teacher-student distillation approach, the student policy learns to fuse proprioception with dual depth streams while remaining robust to real-world sensing imperfections. To further improve robustness, we introduce extensive domain randomization, including stochastic remote-camera dropouts and 3D positional perturbations that emulate aerial-ground cooperative sensing. Simulation results show that multi-viewpoints policies outperform single-viewpoint baseline in gap crossing, step descent, and other dynamic maneuvers, while maintaining stability when the exocentric camera is partially or completely unavailable. Additional experiments show that moderate viewpoint misalignment is well tolerated when incorporated during training. This study demonstrates that heterogeneous visual feedback improves robustness and agility in quadrupedal locomotion. Furthermore, to support reproducibility, the implementation accompanying this work is publicly available at https://anonymous.4open.science/r/multiview-parkour-6FB8

Problem

• Enhances robot environmental awareness using multi-view depth perception.
• Improves locomotion robustness by fusing egocentric and exocentric observations.
• Addresses viewpoint occlusion and sensing imperfections in agile quadrupedal movement.

Innovation

• Multi-view depth fusion for environmental awareness
• Teacher-student distillation with dual depth streams
• Domain randomization for real-world robustness