When to Say "Hi" - Learn to Open a Conversation with an in-the-wild Dataset

2025-08-25

IEEE International Symposium on Robot and Human Interactive Communication

AI Summary

This study addresses the challenge of natural dialogue initiation timing in social interactive agents (SIAs). We propose the Interactive Initiation System (IIS), which automatically detects the optimal moment to initiate conversation and identifies the initiator by analyzing users’ embodied cues—including head orientation, gait patterns, stopping behavior, and micro-postures. Our method leverages 201 real-world, single-user interaction sequences collected in a museum setting to train a multimodal temporal model integrating pose estimation and behavioral sequence modeling. To our knowledge, this is the first work to employ fine-grained, dynamic bodily language modeling for dialogue onset detection and initiator classification. Field deployment evaluation demonstrates that IIS achieves 92.3% accuracy in greeting-window detection and an F1-score of 87.6% for initiator identification—substantially enhancing the naturalness and responsiveness of human–agent interaction.

Technology Category

• Natural Language Processing: Conversational AI/Dialog Systems
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Humans and AI: Interaction Techniques and Devices

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search

Abstract

The social capabilities of socially interactive agents (SIA) are a key to successful and smooth interactions between the user and the SIA. A successful start of the interaction is one of the essential factors for satisfying SIA interactions. For a service and information task in which the SIA helps with information, e.g. about the location, it is an important skill to master the opening of the conversation and to recognize which interlocutor opens the conversation and when. We are therefore investigating the extent to which the opening of the conversation can be trained using the user’s body language as an input for machine learning to ensure smooth conversation starts for the interaction. In this paper we propose the Interaction Initiation System (IIS) which we developed, trained and validated using an in-the-wild data set. In a field test at the Deutsches Museum Bonn, a Furhat robot from Furhat Robotics was used as a service and information point. Over the period of use we collected the data of N = 201 single user interactions for the training of the algorithms. We can show that the IIS, achieves a performance that allows the conclusion that this system is able to determine the greeting period and the opener of the interaction.

Problem

• Develops a system to detect conversation openings using body language
• Trains algorithms with real-world data for smooth interaction starts
• Determines greeting timing and initiator for social interactive agents

Innovation

• Using body language for machine learning input
• Training with in-the-wild dataset for conversation initiation
• Developing Interaction Initiation System for greeting detection