CoMind: Understanding Collaborative Human Activity from Multiple Minds and Views

2026-07-07

AI Summary

Modeling the cognitive mechanisms underlying human collaboration in naturalistic settings—such as theory of mind—remains a significant challenge. To address this, this work introduces CoMind, the first real-world collaborative cooking dataset, which synchronously captures first- and third-person video, audio, eye-tracking data, and 3D scene scans, along with rich annotations of shared attention, social cues, and interactive behaviors. Building on this multimodal resource, we propose three benchmark tasks: joint attention estimation, object interaction prediction under social context, and collaborative handover prediction. By integrating multimodal perception with temporally aligned annotations, CoMind establishes a novel evaluation framework for advancing AI systems’ capabilities in perceiving, reasoning about, and proactively assisting in complex social collaboration.

Abstract

Human-human collaboration is a fundamental aspect of everyday life, essential to success in a wide range of goal-directed activities from household tasks to professional teamwork. While much research has focused on modeling coordination and task execution, the cognitive processes that support such collaboration, particularly Theory of Mind (the ability to infer the mental states of others), remain difficult to study in natural settings. To address this gap, we introduce a novel egocentric and exocentric video dataset capturing real-world collaboration in cooking scenarios. The dataset integrates multi-perspective video, high-quality audio, gaze tracking, and 3D scene and object scans, with annotations for shared attention to objects, social cues and interactions between agents, as well as agent-object interactions. We establish benchmarks for Joint Attention Estimation, Socially Conditioned Object Interaction Anticipation, and Collaborative Handover Prediction, enabling research on multimodal perception, proactive assistance, and collaborative planning. By providing temporally aligned, richly annotated multimodal data, CoMind facilitates the development and evaluation of AI systems capable of modeling complex social interactions and reasoning about human behaviors in collaborative environments. Our dataset and benchmarks are made available at https://comind.ethz.ch/.

Problem

• Theory of Mind
• human collaboration
• multimodal data
• joint attention
• social interaction

Innovation

• Theory of Mind
• multimodal dataset
• joint attention
• collaborative interaction
• egocentric-exocentric vision