Partner Modelling Emerges in Recurrent Agents (But Only When It Matters)

2025-05-22

AI Summary

It remains unclear whether collaborative agents can spontaneously develop internal representations of partner capabilities without explicit other-agent modeling mechanisms. Method: We train model-agnostic RNN agents in the Overcooked-AI environment to cooperate with diverse partners, and analyze their hidden-state dynamics and behavioral trajectories using representational probing techniques. Contribution/Results: We provide the first empirical evidence that structured implicit representations of partner capability spontaneously emerge—but only when task allocation authority is present, i.e., when an agent’s individual decisions substantively influence partner behavior, inducing social pressure. This reveals “collaboration-driven representation learning” as a novel mechanism, wherein social control—rather than architectural specialization—is the critical condition for implicit modeling emergence. Consequently, agents achieve rapid cross-partner adaptation and generalization. Our findings establish a viable, module-free pathway toward flexible collaborative AI systems grounded in emergent social cognition.

Technology Category

• Multiagent Systems: Modeling other Agents
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

Humans are remarkably adept at collaboration, able to infer the strengths and weaknesses of new partners in order to work successfully towards shared goals. To build AI systems with this capability, we must first understand its building blocks: does such flexibility require explicit, dedicated mechanisms for modelling others -- or can it emerge spontaneously from the pressures of open-ended cooperative interaction? To investigate this question, we train simple model-free RNN agents to collaborate with a population of diverse partners. Using the `Overcooked-AI' environment, we collect data from thousands of collaborative teams, and analyse agents' internal hidden states. Despite a lack of additional architectural features, inductive biases, or auxiliary objectives, the agents nevertheless develop structured internal representations of their partners' task abilities, enabling rapid adaptation and generalisation to novel collaborators. We investigated these internal models through probing techniques, and large-scale behavioural analysis. Notably, we find that structured partner modelling emerges when agents can influence partner behaviour by controlling task allocation. Our results show that partner modelling can arise spontaneously in model-free agents -- but only under environmental conditions that impose the right kind of social pressure.

Problem

• Investigates if AI agents can spontaneously model partners during collaboration
• Explores partner modeling emergence in model-free RNN agents without explicit mechanisms
• Identifies environmental conditions enabling spontaneous partner modeling in cooperative tasks

Innovation

• RNN agents model partners without extra mechanisms
• Partner modeling emerges from cooperative interaction pressures
• Probing techniques reveal structured internal representations