GestureCoach: Rehearsing for Engaging Talks with LLM-Driven Gesture Recommendations

2025-04-14

AI Summary

This study addresses the lack of effective guidance for gesture usage in speech rehearsal. We propose a text-driven intelligent gesture prompting method comprising two modules: (1) an LLM-based temporal emphasis detection module that precisely identifies semantic nodes requiring enhanced expressiveness, and (2) a semantics-aware gesture retrieval module integrating expert-annotated gesture libraries with semantic similarity matching to enable personalized, real-time gesture recommendations. The system supports interactive rehearsal with dynamic prompts indicating optimal timing and type of gestures. Our key contribution lies in the first joint modeling of emphasis perception and semantically grounded gesture retrieval—outperforming general-purpose large language models. A user study (N=30) demonstrates superior accuracy in gesture region recommendation; a controlled experiment (N=10) shows that users increased gesture frequency by 42% and gesture diversity by 37%, with independent evaluators rating their presentations as significantly more engaging.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Natural Language Processing: Prompt Engineering / Prompting
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

This paper introduces GestureCoach, a system designed to help speakers deliver more engaging talks by guiding them to gesture effectively during rehearsal. GestureCoach combines an LLM-driven gesture recommendation model with a rehearsal interface that proactively cues speakers to gesture appropriately. Trained on experts' gesturing patterns from TED talks, the model consists of two modules: an emphasis proposal module, which predicts when to gesture by identifying gesture-worthy text segments in the presenter notes, and a gesture identification module, which determines what gesture to use by retrieving semantically appropriate gestures from a curated gesture database. Results of a model performance evaluation and user study (N=30) show that the emphasis proposal module outperforms off-the-shelf LLMs in identifying suitable gesture regions, and that participants rated the majority of these predicted regions and their corresponding gestures as highly appropriate. A subsequent user study (N=10) showed that rehearsing with GestureCoach encouraged speakers to gesture and significantly increased gesture diversity, resulting in more engaging talks. We conclude with design implications for future AI-driven rehearsal systems.

Problem

• Helps speakers deliver engaging talks with gestures
• Uses LLM to recommend effective gestures during rehearsal
• Improves gesture diversity and talk engagement

Innovation

• LLM-driven gesture recommendation model
• Emphasis proposal module predicts gesture timing
• Gesture identification module retrieves appropriate gestures