Interaction Analysis by Humans and AI: A Comparative Perspective

2025-06-09

AI Summary

This study investigates how mixed reality (MR; HoloLens) and 2D video conferencing (Zoom) affect children’s communication during a collaborative gesture-based word-guessing game. To address the challenge of analyzing non-English, multimodal child interaction data across platforms, we introduce an LLM-driven automated framework for transcription, translation, and annotation—enabling cross-platform comparative analysis and facilitating efficient participation by non-native-speaking researchers. Results show that MR significantly enhances children’s affective expressiveness and task engagement, whereas Zoom excels in usability and accessibility. While LLM-based annotation exhibits moderate accuracy, it sufficiently supports robust cross-platform comparisons and reduces data processing time substantially. This work pioneers the integration of LLMs into multimodal collaborative research with children, providing empirical evidence for MR in educational contexts and establishing a reproducible methodological paradigm. (149 words)

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

This paper explores how Mixed Reality (MR) and 2D video conferencing influence children's communication during a gesture-based guessing game. Finnish-speaking participants engaged in a short collaborative task using two different setups: Microsoft HoloLens MR and Zoom. Audio-video recordings were transcribed and analyzed using Large Language Models (LLMs), enabling iterative correction, translation, and annotation. Despite limitations in annotations' accuracy and agreement, automated approaches significantly reduced processing time and allowed non-Finnish-speaking researchers to participate in data analysis. Evaluations highlight both the efficiency and constraints of LLM-based analyses for capturing children's interactions across these platforms. Initial findings indicate that MR fosters richer interaction, evidenced by higher emotional expression during annotation, and heightened engagement, while Zoom offers simplicity and accessibility. This study underscores the potential of MR to enhance collaborative learning experiences for children in distributed settings.

Problem

• Compare MR and Zoom effects on children's communication
• Evaluate LLM efficiency in analyzing gesture-based interactions
• Assess MR's potential to enhance collaborative learning

Innovation

• Mixed Reality enhances children's interaction
• LLMs enable multilingual data analysis
• Automated processing reduces analysis time