Watch Your Step: Learning Semantically-Guided Locomotion in Cluttered Environment

2026-03-03

AI Summary

This work addresses the challenge of legged robots inadvertently stepping on low-lying, sensitive objects—such as cables or equipment—in cluttered environments due to a lack of semantic awareness. To this end, the authors propose SemLoco, a novel framework that integrates semantic maps with reinforcement learning for foothold planning. SemLoco employs a two-stage reinforcement learning strategy that combines hard and soft constraints to infer safe footholds on pixel-level elevation maps, while leveraging semantic information to construct a semantics-driven traversability cost. Experimental results demonstrate that SemLoco significantly reduces collision rates with sensitive objects and enables reliable, safe navigation in real-world complex environments, outperforming conventional controllers.

Technology Category

• Intelligent Robots: Localization, Mapping, and Navigation
• Natural Language Processing: Safety and Robustness
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search
• Systems and Infrastructure for Web, Mobile and WoT: Location- and context-aware Web and WoT applications and services

Abstract

Although legged robots demonstrate impressive mobility on rough terrain, using them safely in cluttered environments remains a challenge. A key issue is their inability to avoid stepping on low-lying objects, such as high-cost small devices or cables on flat ground. This limitation arises from a disconnection between high-level semantic understanding and low-level control, combined with errors in elevation maps during real-world operation. To address this, we introduce SemLoco, a Reinforcement Learning (RL) framework designed to avoid obstacles precisely in densely cluttered environments. SemLoco uses a two-stage RL approach that combines both soft and hard constraints and performs pixel-wise foothold safety inference, enabling more accurate foot placement. Additionally, SemLoco integrates a semantic map to assign traversability costs rather than relying solely on geometric data. SemLoco significantly reduces collisions and improves safety around sensitive objects, enabling reliable navigation in situations where traditional controllers would likely cause damage. Experimental results further demonstrate that SemLoco can be effectively applied to more complex, unstructured real-world environments.

Problem

• legged robots
• cluttered environments
• semantic understanding
• foot placement
• obstacle avoidance

Innovation

• SemLoco
• semantic mapping
• reinforcement learning
• foot placement
• legged locomotion