ViBe: Visual Behavior Adaptation for Perceptive Humanoid Whole-Body Control

2026-09-09

AI Summary

本文提出ViBe框架，通过预训练视觉编码器和低秩适配器改进运动追踪器，以适应感知控制任务，实现零样本仿真到现实的迁移。

Abstract

Motion tracking provides a scalable recipe for humanoid whole-body control. By design, the resulting trackers lack exteroceptive feedback hence reacting to the environment remains the responsibility of a higher-level planner. Existing perceptive controllers train geometry-only encoders from scratch, trading semantics for sim-to-real ease, and typically rely on teacher-student distillation for a task of interest. We present ViBe, a post-training framework for adapting motion trackers to perceptive control tasks. We leverage pre-trained visual encoders with a multi-query extractor module to learn task-relevant perceptive feedback. This feedback is grafted onto the tracker's input via low-rank adapters, enabling parameter-efficient fine-tuning. Given a task reward and a reference dataset, this modular controller can be adapted directly via policy optimization. Across four tasks, ViBe shows zero-shot sim-to-real transfer spanning perceptive walking on curbs and parkour, Repose Cube, omni-object loco-manipulation, and dodgeball, with visually robust performance across outdoor, low-light, and RGB distractor conditions. Finally, we solve a goal-oriented Repose Cube task with a deliberately simple planner, demonstrating the efficacy of perceptive controllers, adapted by our approach.

Problem

• motion tracking
• perceptive control
• visual behavior adaptation
• humanoid whole-body control
• environmental feedback

Innovation

• perceptive control
• visual behavior adaptation
• low-rank adapters
• parameter-efficient fine-tuning
• zero-shot sim-to-real transfer