LLM-Based Synthetic Ground Truth Generation for Audio-Based Emotion Classification via In-Context Learning

2026-06-10

AI Summary

This work addresses the challenge of dynamically inferring team affective states in multi-user virtual reality environments, where the scarcity of high-quality annotated data and the limitations of traditional self-reporting methods hinder real-time emotion capture during interactions. To overcome this, the authors propose a fine-tuning-free approach leveraging large language models (LLMs): acoustic feature similarity is used to retrieve contextual examples, which—combined with a few annotated samples—enable in-context learning (ICL) to automatically generate emotion-related synthetic ground-truth labels from streaming speech. The method achieves performance comparable to fine-tuned models on multimodal affective analysis tasks while substantially reducing computational overhead, offering a highly efficient and practical synthetic labeling solution for continuous speech-based emotion classification.

Abstract

Understanding human states and interaction dynamics is a core goal of human-computer interaction (HCI). As interaction paradigms become more immersive, virtual reality (VR) has emerged as a powerful platform for studying collaborative work. In such settings, evaluating team collaboration states, including team performance and team resilience, requires continuous and reliable inference of latent team-level cognitive and affective states from multi-modal sensor data, such as speech signals. However, generating ground truth labels for these latent states remains challenging due to sensor-induced noise, contextual variability, and sparse expert annotations. Traditional self-reporting approaches provide only static and delayed measurements and are therefore insufficient for capturing dynamic team processes reflected in continuous speech data. In this work, we propose a large language model (LLM)-driven, agentic inference workflow for automated emotion-related synthetic ground truth generation from streaming speech data in multi-user VR environments. Leveraging the generalization capabilities of LLMs, we use In-Context Learning (ICL) with few-shot demonstrations of paired audio-based samples and their corresponding transcriptions. ICL tends to achieve task adaptation comparable to model fine-tuning while circumventing the computational overhead of parameter updates. To construct informative and robust in-context prompts, we adopt a retrieval-based selection strategy that dynamically identifies relevant audio demonstrations based on similarity in the acoustic feature space.

Problem

• emotion classification
• ground truth generation
• team collaboration
• virtual reality
• speech signals

Innovation

• In-Context Learning
• Synthetic Ground Truth
• Audio-Based Emotion Classification
• Large Language Models
• Retrieval-Based Prompting