Mind & Motion: Opportunities and Applications of Integrating Biomechanics and Cognitive Models in HCI

2025-08-19

AI Summary

This study addresses the fragmentation in user behavior modeling for human–computer interaction by proposing the first systematic multimodal prediction framework that unifies cognitive and biomechanical models. Methodologically, it integrates computational cognitive modeling, high-fidelity biomechanical simulation, human factors engineering, and behavioral data analysis to jointly model user intent recognition, strategy selection, and motor execution across diverse user populations. Its key contribution lies in transcending the traditional “cognition–body” dichotomy by introducing an embodied user simulation model, thereby significantly enhancing the fidelity and generalizability of behavioral predictions. The framework provides an interpretable, quantifiable benchmark modeling tool for interface design optimization, ergonomic assessment, automated interaction testing, and personalized adaptive systems—advancing UI/UX evaluation from experience-driven to computation-driven paradigms.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Humans and AI: Intelligent User Interfaces
• Application Domains: Humanities & Computational Social Science

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Computational models of how users perceive and act within a virtual or physical environment offer enormous potential for the understanding and design of user interactions. Cognition models have been used to understand the role of attention and individual preferences and beliefs on human decision making during interaction, while biomechanical simulations have been successfully applied to analyse and predict physical effort, fatigue, and discomfort. The next frontier in HCI lies in connecting these models to enable robust, diverse, and representative simulations of different user groups. These embodied user simulations could predict user intents, strategies, and movements during interaction more accurately, benchmark interfaces and interaction techniques in terms of performance and ergonomics, and guide adaptive system design. This UIST workshop explores ideas for integrating computational models into HCI and discusses use cases such as UI/UX design, automated system testing, and personalised adaptive interfaces. It brings researchers from relevant disciplines together to identify key opportunities and challenges as well as feasible next steps for bridging mind and motion to simulate interactive user behaviour.

Problem

• Integrating biomechanics and cognitive models for user simulations
• Predicting user intents, strategies, and movements during interaction
• Benchmarking interfaces for performance and ergonomics

Innovation

• Integrating biomechanics and cognitive models
• Simulating user behavior for adaptive design
• Bridging mind and motion in HCI