CART: Context-Aware Terrain Adaptation using Temporal Sequence Selection for Legged Robots

2026-04-15

AI Summary

This work addresses the “visual–textural paradox” in legged robots operating in complex natural terrains—a discrepancy between visual perception and actual physical interaction caused by overreliance on visual priors. To mitigate this issue, the authors propose a context-aware terrain-adaptive control framework that fuses proprioceptive and multimodal exteroceptive sensing. By dynamically selecting salient temporal segments to infer terrain properties in real time, the method constructs a high-level motion controller optimized for body vibration stability. The novel context-aware mechanism and temporal selection strategy effectively alleviate the visual–textural paradox, yielding a 5% improvement in task success rate in simulation and enhancing stability by 45% and 24% in real-world experiments, without increasing task duration.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Low Level & Physics-based Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Location- and context-aware Web and WoT applications and services

Abstract

Animals in nature combine multiple modalities, such as sight and feel, to perceive terrain and develop an understanding of how to walk on uneven terrain in a stable manner. Similarly, legged robots need to develop their ability to stably walk on complex terrains by developing an understanding of the relationship between vision and proprioception. Most current terrain adaptation methods are susceptible to failure on complex, off-road terrain as they rely on prior experience, particularly observations from a vision sensor. This experience-based learning often creates a Visual-Texture Paradox between what has been seen and how it actually feels. In this work, we introduce CART, a high-level controller built on a context-aware terrain adaptation approach that integrates proprioception and exteroception from onboard sensing to achieve a robust understanding of terrain. We evaluate our method on multiple terrains using an ANYmal-C robot on the IsaacSim simulator and a Boston Dynamics SPOT robot for our real-world experiments. To evaluate the learned contextual terrain properties, we adapt vibrational stability on the base of the robot as a metric. We compare CART with various state-of-the-art baselines equipped with multimodal sensing in both simulation and the real world. CART achieves an average success rate improvement of 5% over all baselines in simulation and improves the overall stability up to 45% and 24% in the real world without increasing the time taken by the robot to accomplish locomotion tasks.

Problem

• terrain adaptation
• legged robots
• visual-texture paradox
• proprioception
• exteroception

Innovation

• context-aware adaptation
• multimodal sensing
• proprioception-exteroception fusion
• terrain understanding
• legged locomotion