AME-2: Agile and Generalized Legged Locomotion via Attention-Based Neural Map Encoding

2026-01-13

AI Summary

This work proposes a unified reinforcement learning framework to address the challenges of generalization and interpretability in locomotion control for legged robots operating in complex, occluded, and sparsely feasible foothold terrains. By leveraging an attention mechanism to fuse local and global map features, the approach generates interpretable and generalizable control embeddings. It tightly couples perception and control through a learned, noise-robust, and uncertainty-aware online mapping pipeline. The framework innovatively integrates neural map construction, depth-to-elevation estimation, odometry fusion, and parallel simulation-based training. Extensive evaluations on both quadrupedal and bipedal platforms demonstrate that the resulting controller achieves exceptional agility and strong generalization to previously unseen terrains in both simulation and real-world environments.

Technology Category

• Intelligent Robots: Localization, Mapping, and Navigation
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Achieving agile and generalized legged locomotion across terrains requires tight integration of perception and control, especially under occlusions and sparse footholds. Existing methods have demonstrated agility on parkour courses but often rely on end-to-end sensorimotor models with limited generalization and interpretability. By contrast, methods targeting generalized locomotion typically exhibit limited agility and struggle with visual occlusions. We introduce AME-2, a unified reinforcement learning (RL) framework for agile and generalized locomotion that incorporates a novel attention-based map encoder in the control policy. This encoder extracts local and global mapping features and uses attention mechanisms to focus on salient regions, producing an interpretable and generalized embedding for RL-based control. We further propose a learning-based mapping pipeline that provides fast, uncertainty-aware terrain representations robust to noise and occlusions, serving as policy inputs. It uses neural networks to convert depth observations into local elevations with uncertainties, and fuses them with odometry. The pipeline also integrates with parallel simulation so that we can train controllers with online mapping, aiding sim-to-real transfer. We validate AME-2 with the proposed mapping pipeline on a quadruped and a biped robot, and the resulting controllers demonstrate strong agility and generalization to unseen terrains in simulation and in real-world experiments.

Problem

• legged locomotion
• agility
• generalization
• visual occlusions
• sparse footholds

Innovation

• attention-based map encoding
• generalized locomotion
• uncertainty-aware mapping
• sim-to-real transfer
• legged robotics