What Do You Think I Think? Accounting for Human Beliefs Using Second-Order Theory of Mind

2026-05-12

AI Summary

This work addresses the challenge in human-agent collaboration where human cognitive biases often lead to inaccurate beliefs about the agent’s knowledge state, thereby impeding effective coordination. The paper presents the first approach that integrates second-order theory of mind (ToM-2) with explicit modeling of cognitive biases within an interactive partially observable Markov decision process (I-POMDP) framework. This enables the agent to infer not only the human’s mistaken beliefs about its own mental state but also the underlying heuristic mechanisms driving those biases, allowing it to generate targeted, adaptive feedback. Experimental results demonstrate that the proposed method significantly increases the informational value of teaching actions provided by the agent, and user evaluations confirm that the generated feedback is perceived as more useful, offering a novel pathway toward cognitively aligned human-agent collaboration.

Abstract

Discrepancies between an agent's actual knowledge and what a person thinks the agent knows can hinder interactions. If an agent could detect such discrepancies, it could provide feedback to account for them and improve current and future interactions. Using the I-POMDP as a framework for a second-order Theory of Mind (ToM-2), this work endows an agent with the ability to model the evolution of a person's erroneous beliefs about an agent and the cognitive biases and heuristics (CBH) from which they arise. In doing so, the agent can detect when CBH might be at play during an interaction and adaptively generate feedback that accounts for them. An in-person user study shows how a ToM-2 learner can account for the effects of a teacher's CBH to significantly improve the informativeness of teacher actions, and subjective results suggest people find the ToM-2 learner's feedback more useful.

Problem

• Theory of Mind
• cognitive biases
• human-agent interaction
• belief discrepancy
• second-order reasoning

Innovation

• second-order Theory of Mind
• cognitive biases and heuristics
• I-POMDP
• belief modeling
• adaptive feedback