Embodied Explainability and Ontological Obstacles: Why We Struggle to Explain the Answers of Large Language Models (LLMs)

2026-06-22

AI Summary

This study addresses a critical limitation in current explainable artificial intelligence (XAI), which treats explanations as intrinsic properties of models while neglecting the active, situated process through which humans co-construct understanding during interaction. This oversight leads to ontological misalignment and risks of user overreliance, particularly with large language models. Drawing on Dourish’s theory of embodied interaction and enactive cognition, the paper introduces the concept of “embodied explainability,” arguing that explanations should emerge through interactive opportunities that support probing, coordination, and repair within specific contexts, thereby enabling the co-construction of shared meaning rather than merely revealing internal model mechanisms. Through philosophical argumentation, conceptual analysis, and illustrative examples, this work critiques XAI’s problematic conflation of explanation with model transparency and offers a novel theoretical foundation for designing explainability systems grounded in practical, human-centered interaction.

Abstract

Explainability is often framed as a property of an AI model, with explanations extracted from its internals and shown to users. In this argument paper, we instead provide an embodied account of explainability based on Dourish and enactivist cognition: understanding is created in use as people act on affordances in shared practice. Using demonstrations and conceptual analysis, we reveal ontological obstacles when "looking inside" large language models: surrogates import external abstractions that can be mistaken for the model's, and focusing on internal reasoning misses that explainers participate in their own understanding. We discuss these obstacles in XAI practice, arguing that many explanations are misnamed, which skews their purpose and can increase overreliance. Finally, we highlight how embodied explanations reorganize sense-making by making what matters publicly available for action, and argue that explainability claims should be reserved for designs that provide affordances to probe, coordinate, and repair behaviour in situated practice.

Problem

• Explainability
• Large Language Models
• Ontological Obstacles
• Embodied Cognition
• Human-AI Interaction

Innovation

• embodied explainability
• ontological obstacles
• enactivist cognition
• affordances
• situated practice