🤖 AI Summary
This work addresses the challenge of achieving generalizable and stable behavioral alignment for embodied agents operating in 3D visual environments. We propose the first framework that systematically adapts large language model alignment paradigms—such as Reinforcement Learning from Human Feedback (RLHF)—to pixel-level embodied agents. Our method integrates behavior cloning for initialization, multimodal contrastive preference modeling, and pixel-input-based Proximal Policy Optimization (PPO) for reinforcement fine-tuning, enabling fine-grained alignment over behavioral modality selection without hand-crafted reward functions. In experiments within a game-based sub-environment, our agent achieves 100% focus accuracy on specified behavioral modes, substantially outperforming all baselines. Furthermore, we provide a reusable alignment pipeline and practical implementation guidelines. This work establishes a novel pathway toward general behavioral alignment for vision-driven embodied agents.
📝 Abstract
Training agents to behave as desired in complex 3D environments from high-dimensional sensory information is challenging. Imitation learning from diverse human behavior provides a scalable approach for training an agent with a sensible behavioral prior, but such an agent may not perform the specific behaviors of interest when deployed. To address this issue, we draw an analogy between the undesirable behaviors of imitation learning agents and the unhelpful responses of unaligned large language models (LLMs). We then investigate how the procedure for aligning LLMs can be applied to aligning agents in a 3D environment from pixels. For our analysis, we utilize an academically illustrative part of a modern console game in which the human behavior distribution is multi-modal, but we want our agent to imitate a single mode of this behavior. We demonstrate that we can align our agent to consistently perform the desired mode, while providing insights and advice for successfully applying this approach to training agents. Project webpage at https://adamjelley.github.io/aligning-agents-like-llms .