TANDE: Disentangling Verbal and Nonverbal Backchannels in Emotional AI-Avatar Conversations with Young Adults

2026-07-14

AI Summary

This study addresses the limited understanding of how verbal and nonverbal feedback channels in AI-driven virtual characters influence emotional interaction, particularly regarding gender differences and emotional support efficacy among young users. The authors developed TANDE, an empathetic conversational AI agent powered by a large language model, and for the first time decoupled verbal, nonverbal, and multimodal feedback channels. Through a within-subjects experimental design (N=36), they quantitatively assessed the impact of these channels on perceived empathy, intimacy, and engagement. Results reveal that young users exhibit a significant preference for nonverbal cues, which also demonstrate superior effectiveness in enhancing empathy and overall interaction quality. These findings provide empirical grounding and multimodal interaction design guidelines for developing AI systems aimed at supporting adolescent mental health.

Abstract

Embodied conversational agents (ECAs) need effective empathic grounding to foster social support and engagement. Expanding into emotional domains, ECAs now use Large Language Models (LLMs) and multimodal human-agent interactions to enhance their capabilities. Yet, understanding the impact of backchanneling modalities on young adults and their gender remains limited. We introduce TANDE, an LLM-powered ECA designed for emotional conversations with young adults, a population experiencing mental, personal, and social issues with limited tools to address them. In a within-subjects study with N=36 young adults, we explore nonverbal and combined verbal-and-nonverbal backchanneling modalities on rapport, empathy, and engagement and isolate for gender differences. Our research shows the importance of nuanced backchanneling cues with emotional ECAs with young adults, showing a preference for nonverbal cues. We derive design implications for more effective ECAs for emotional support and well-being in young adults. The code is available at https://github.com/Cornell-Tech-AIRLab/TANDE.

Problem

• backchanneling
• embodied conversational agents
• emotional AI
• young adults
• gender differences

Innovation

• backchanneling
• embodied conversational agents
• multimodal interaction
• emotional AI
• young adults