Do as the Romans Do: Learning Universal Behaviors from Heterogeneous Agents

2026-06-16

AI Summary

This work addresses the challenge that conventional imitation learning struggles to extract generalizable behaviors from heterogeneous agents with diverse objectives, often suffering from conflicting behavioral signals and mode-averaging bias. To overcome this limitation, the paper introduces GRID, a novel approach that decouples demonstrators’ reward functions—via reward decomposition—into semantically meaningful universal and agent-specific components. GRID establishes a new paradigm for generalist pretraining by performing unsupervised social learning solely on the universal reward component. Experimental results demonstrate that GRID effectively disentangles reward structures in environments such as Craftax and Highway-Env, significantly outperforming standard imitation learning baselines and substantially improving both the efficiency and stability of downstream task fine-tuning.

Abstract

Humans often acquire new skills by observing others, since observed behaviors implicitly reveal how to act in an environment. However, observations drawn from a heterogeneous population introduce conflicting behavioral signals, making it difficult to determine which behaviors are worth imitating. We address this challenge with General Reward Inference and Disentanglement (GRID), a social learning method that extracts universally useful behaviors from a heterogeneous population of demonstrators pursuing different goals. GRID decomposes per-agent reward functions into a general reward, capturing behaviors shared across all agents, and specific rewards, capturing individual preferences and objectives. Training exclusively on the general reward provides a new paradigm of generalist pretraining. It yields a generalist agent that internalizes universal environmental competencies, such as safety and basic task proficiency, without the mode-averaging bias that afflicts standard learning from demonstration techniques. This generalist serves as a superior prior for fine-tuning to downstream tasks, including preferences unseen during training. Experiments across a synthetic basis function decomposition, multi-agent Craftax, and a continuous autonomous driving simulator (Highway-Env) confirm that GRID successfully disentangles reward structure in a semantically meaningful way, outperforms standard learning from demonstration baselines, and enables more efficient and stable specialization.

Problem

• social learning
• heterogeneous agents
• universal behaviors
• reward disentanglement
• learning from demonstration

Innovation

• reward disentanglement
• social learning
• generalist pretraining
• heterogeneous demonstrators
• imitation learning