Situated, Dynamic, and Subjective: Envisioning the Design of Theory-of-Mind-Enabled Everyday AI with Industry Practitioners

2026-02-11

AI Summary

This study addresses the lack of viable design pathways for integrating theory of mind (ToM) into everyday AI products. Through 13 co-design workshops involving 26 U.S.-based AI practitioners, the research explores how ToM-enabled AI systems can interpret user intentions within social contexts and support sustained interaction. The work proposes three core design principles—contextuality, dynamism, and subjectivity—that move beyond traditional static reasoning paradigms, positioning ToM as a foundational capability throughout the human-AI interaction lifecycle. Drawing on co-design sessions, in-depth interviews, and thematic analysis, the study distills three actionable design recommendations grounded in real-world development scenarios, revealing tensions between practitioners’ aspirational visions and current technical constraints, thereby offering both theoretical insights and practical guidance for future AI product development.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Philosophy and Ethics of AI: Accountability, Interpretability & Explainability

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Psychology-informed user models and recommender systems
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems

Abstract

Theory of Mind (ToM) -- the ability to infer what others are thinking (e.g., intentions) from observable cues -- is traditionally considered fundamental to human social interactions. This has sparked growing efforts in building and benchmarking AI's ToM capability, yet little is known about how such capability could translate into the design and experience of everyday user-facing AI products and services. We conducted 13 co-design sessions with 26 U.S.-based AI practitioners to envision, reflect, and distill design recommendations for ToM-enabled everyday AI products and services that are both future-looking and grounded in the realities of AI design and development practices. Analysis revealed three interrelated design recommendations: ToM-enabled AI should 1) be situated in the social context that shape users'mental states, 2) be responsive to the dynamic nature of mental states, and 3) be attuned to subjective individual differences. We surface design tensions within each recommendation that reveal a broader gap between practitioners'envisioned futures of ToM-enabled AI and the realities of current AI design and development practices. These findings point toward the need to move beyond static, inference-driven approach to ToM and toward designing ToM as a pervasive capability that supports continuous human-AI interaction loops.

Problem

• Theory of Mind
• AI design
• human-AI interaction
• everyday AI
• mental states

Innovation

• Theory of Mind
• co-design
• situated AI
• dynamic mental states
• subjective individual differences