Does the TalkMoves Codebook Generalize to One-on-One Tutoring and Multimodal Interaction?

2026-04-14

AI Summary

This study presents the first systematic evaluation of the generalizability of the classic classroom discourse coding scheme TalkMoves in one-on-one tutoring and multimodal interaction settings. Through expert annotation, multimodal data analysis (text, audio, and video), and Cohen’s kappa inter-rater reliability assessment, the authors compare TalkMoves with a hybrid coding scheme co-developed by AI and human annotators in terms of reliability, coverage, and multimodal applicability. Results indicate that while TalkMoves achieves higher overall inter-rater agreement (κ = 0.74), it struggles to effectively capture nonverbal cues. In contrast, the hybrid coding scheme demonstrates broader coverage and superior adaptability across modalities. These findings highlight the limitations of existing coding frameworks in tutoring contexts and provide theoretical grounding and practical guidance for designing next-generation behavioral coding frameworks tailored to multimodal tutoring interactions.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Accountable Talk theory has been widely adopted to analyze classroom discourse and is increasingly used to annotate tutoring interactions. In particular, the TalkMoves codebook, grounded in Accountable Talk theory, is commonly used to label tutoring data and train models of effective instructional support. However, Accountable Talk was originally developed to characterize collaborative, whole-classroom oral discourse, not to identify talk moves in one-on-one tutoring environments using multimodal data (e.g., video, audio, chat). As tutoring platforms expand in scale and modality, questions remain about whether Accountable Talk-based codebooks generalize reliably beyond their original classroom context and data representation. This study examines whether the human-developed TalkMoves codebook generalizes in reliability, utility, and interpretability when applied to one-on-one tutoring across audio, chat, and multimodal data. We compare TalkMoves with a hybrid AI-human developed codebook using a workflow established in prior research. Two expert annotators with over 20 years of teaching experience applied both codebooks to six tutoring sessions spanning three modalities: chat-based, audio-only, and multimodal interactions. Results show that while Talk-Moves achieved higher overall inter-rater reliability than the AI-human codebook (k = 0.74 vs. 0.64), the AI-human codebook demonstrated broader empirical coverage and higher perceived usability across modalities. Both codebooks undercaptured tutoring-relevant moves and introduced ambiguity when identifying actions expressed through nonverbal and multimodal artifacts. Together, these findings highlight the uneven generalizability of TalkMoves to tutoring contexts and motivate the development of modality-aware, tutoring-grounded codebooks.

Problem

• TalkMoves
• Accountable Talk
• one-on-one tutoring
• multimodal interaction
• codebook generalizability

Innovation

• TalkMoves
• Accountable Talk
• multimodal tutoring
• codebook generalization
• AI-human hybrid annotation