DigitalCoach: Communication and Grounding Gaps in Human and Agentic Computer Use Coaching

2026-06-30

AI Summary

This study addresses the limitations of current AI agents in software instruction, particularly their communication gaps and disconnection from visual context. The authors introduce DigitalCoach, a multimodal dataset comprising 72 expert tutoring sessions, 22,752 dialogue turns, and 28.1 hours of synchronized screen and input recordings, which enables the first systematic evaluation of large language models in explanation, error diagnosis, and knowledge assessment tasks. Combining multimodal data collection, automated metrics, and user studies, the work reveals that while models can generate human-like instructions, they often fail to ground their responses in the visual context, frequently issuing superficial directives that encourage passive execution rather than active learning. This research lays the groundwork for developing collaborative, proactive AI tutors capable of context-aware software instruction.

Abstract

Agents are increasingly capable of automating software tasks, but can they teach humans how to use software themselves? We introduce DigitalCoach, a multimodal dataset of 72 human expert-novice computer use coaching sessions consisting of 22,752 dialogue turns grounded in 28.1 hours of screen and input event recordings across five software applications. We use DigitalCoach to evaluate whether state-of-the-art models can teach humans how to use computers. Automated evaluation shows that models differ from humans in how they coach: models provide more direct instructions, but fewer explanations, error diagnoses, and knowledge-check questions. When we fix the coaching method, models produce utterances similar to human references yet poorly grounded in visual context. Interactive evaluation confirms that model coaches cause learners to passively follow instructions without deeper engagement and fall short in visual grounding. DigitalCoach lays a foundation for collaborative and proactive computer use coaching agents.

Problem

• computer use coaching
• visual grounding
• human-agent communication
• instructional dialogue
• multimodal interaction

Innovation

• multimodal dataset
• computer use coaching
• visual grounding
• human-AI collaboration
• instructional dialogue