Understanding Human Actions through the Lens of Executable Models

2026-04-20

AI Summary

Existing action recognition methods struggle to model the intrinsic execution structure of human actions, limiting their ability to assess action quality and distinguish fine-grained variations. This work proposes an executable neuro-symbolic model that represents actions as under-specified motor programs using the domain-specific language EXACT, enabling zero-shot policy inference through bidirectional representations. For the first time, the internal execution mechanism of actions is explicitly modeled in a procedural and composable manner, facilitating structured understanding and generalization. The approach significantly improves data efficiency in action segmentation and anomaly detection tasks, captures semantic relationships among actions, and outperforms conventional monolithic methods.

Technology Category

• Machine Learning: Neuro-Symbolic Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Human-centred systems require an understanding of human actions in the physical world. Temporally extended sequences of actions are intentional and structured, yet existing methods for recognising what actions are performed often do not attempt to capture their structure, particularly how the actions are executed. This, however, is crucial for assessing the quality of the action's execution and its differences from other actions. To capture the internal mechanics of actions, we introduce a domain-specific language EXACT that represents human motions as underspecified motion programs, interpreted as reward-generating functions for zero-shot policy inference using forward-backwards representations. By leveraging the compositional nature of EXACT motion programs, we combine individual policies into an executable neuro-symbolic model that uses program structure for compositional modelling. We evaluate the utility of the proposed pipeline for creating executable action models by analysing motion-capture data to understand human actions, for the tasks of human action segmentation and action anomaly detection. Our results show that the use of executable action models improves data efficiency and captures intuitive relationships between actions compared with monolithic, task-specific approaches.

Problem

• human action recognition
• action execution structure
• executable models
• motion understanding
• action quality assessment

Innovation

• executable models
• neuro-symbolic
• zero-shot policy inference
• compositional modeling
• action understanding