Teaching Robots to Interpret Social Interactions through Lexically-guided Dynamic Graph Learning

2026-04-12

AI Summary

This work proposes SocialLDG, a novel framework aimed at endowing robots with social intelligence by enabling them to infer users’ internal states from observed behaviors, predict future actions, and generate appropriate responses. Integrating insights from cognitive science, SocialLDG uniquely combines lexical priors derived from language models with dynamic graph neural networks to explicitly model the temporally evolving relationships among six core tasks, thereby capturing the dynamic mapping between latent internal states and observable behaviors. The framework supports continual learning of new tasks without catastrophic forgetting and offers strong interpretability and scalability. Evaluated on two public human–robot social interaction datasets, SocialLDG achieves state-of-the-art performance, revealing the underlying temporal dynamics governing the interplay between internal states and behavioral expressions in social interactions.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

For a robot to be called socially intelligent, it must be able to infer users internal states from their current behaviour, predict the users future behaviour, and if required, respond appropriately. In this work, we investigate how robots can be endowed with such social intelligence by modelling the dynamic relationship between user's internal states (latent) and actions (observable state). Our premise is that these states arise from the same underlying socio-cognitive process and influence each other dynamically. Drawing inspiration from theories in Cognitive Science, we propose a novel multi-task learning framework, termed as \textbf{SocialLDG} that explicitly models the dynamic relationship among the states represent as six distinct tasks. Our framework uses a language model to introduce lexical priors for each task and employs dynamic graph learning to model task affinity evolving with time. SocialLDG has three advantages: First, it achieves state-of-the-art performance on two challenging human-robot social interaction datasets available publicly. Second, it supports strong task scalability by learning new tasks seamlessly without catastrophic forgetting. Finally, benefiting from explicit modelling task affinity, it offers insights on how different interactions unfolds in time and how the internal states and observable actions influence each other in human decision making.

Problem

• social intelligence
• internal states
• behavior prediction
• human-robot interaction
• dynamic relationships

Innovation

• dynamic graph learning
• lexical priors
• multi-task learning
• social intelligence
• task affinity