TeamLLM: Exploring the Capabilities of LLMs for Multimodal Group Interaction Prediction

2026-04-09

AI Summary

This study investigates the use of large language models (LLMs) to predict collective interaction behaviors—such as coordination, communication, and interaction patterns—in collaborative teams from multimodal sensor data. By encoding individual profiles, group structure, and temporal context into natural language, the work evaluates the efficacy of zero-shot, few-shot, and supervised fine-tuning paradigms within a mixed-reality environment. It demonstrates for the first time that purely text-based LLMs can effectively model turn-taking dynamics, achieving 96% accuracy after fine-tuning—3.2× higher than LSTM baselines—with latency under 35 ms. However, such models struggle to capture joint attention behaviors that rely on spatial visual reasoning. Additionally, simulation-based inference suffers an 83% performance drop due to cascading contextual errors.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Predicting group behavior, how individuals coordinate, communicate, and interact during collaborative tasks, is essential for designing systems that can support team performance through real-time prediction and realistic simulation of collaborative scenarios. Large Language Models (LLMs) have shown promise for processing sensor data for human-activity recognition (HAR), yet their capabilities for team dynamics or group-level multimodal sensing remain unexplored. This paper investigates whether LLMs can predict group coordination patterns from multimodal sensor data in collaborative Mixed Reality (MR) environments. We encode hierarchical context -- individual behavioral profiles, group structural properties, and temporal activity context -- as natural language and evaluate three LLM adaptation paradigms (zero-shot, few-shot, and supervised fine-tuning) against statistical baselines. Our evaluation on 16 groups (64 participants, $\sim$25 hours of sensor data) reveals that LLMs achieve 3.2$\times$ improvement over LSTM baselines for linguistically-grounded behaviors, with fine-tuning reaching 96\% accuracy for conversation prediction while maintaining sub-35ms latency. Beyond performance gains, we characterize the boundaries of text-based LLMs for multimodal sensing conversation prediction succeeds because turn-taking maps to linguistic patterns, while shared or joint attention may require spatial and visual reasoning that text only LLMs cannot capture. We further identify simulation mode brittleness (83\% degradation from cascading context errors) and minimal few-shot sensitivity to example selection strategy. These findings establish guidelines when LLMs are appropriate for CPS/IoT sensing for team dynamics and inform the design of future multimodal foundation models.

Problem

• group interaction prediction
• multimodal sensing
• team dynamics
• Large Language Models
• collaborative Mixed Reality

Innovation

• Large Language Models
• Multimodal Sensing
• Group Interaction Prediction
• Mixed Reality
• Team Dynamics