HOICraft: In-Situ VLM-based Authoring Tool for Part-Level Hand-Object Interaction Design in VR

2026-02-09

AI Summary

This work addresses the inefficiency of manual trial-and-error approaches commonly used in designing hand-object interactions in virtual reality (VR), which often hinder rapid identification of interactive components and their behavior mappings. To overcome this limitation, the paper introduces HOICraft—the first tool to integrate vision-language models (VLMs) into in-situ VR authoring for component-level interaction design. By synergistically combining VLMs, 3D semantic parsing, and in-context learning, HOICraft automatically recommends interactive elements and maps hand gestures to object behaviors. User studies demonstrate that HOICraft significantly reduces the number of trial-and-error iterations and outperforms conventional manual design in both usability and effectiveness, establishing a novel and efficient paradigm for VR interaction authoring.

Technology Category

• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Computer Vision: Language and Vision
• Intelligent Robots: Human-Robot Interaction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Hand-Object Interaction (HOI) is a key interaction component in Virtual Reality (VR). However, designing HOI still requires manual efforts to decide how object should be selected and manipulated, while also considering user abilities, which leads to time-consuming refinements. We present HOICraft, a VLM-based in-situ HOI authoring tool that enables part-level interaction design in VR. Here, HOICraft assists designers by recommending interactable elements from 3D objects, customizing HOI design properties, and mapping hand movement with virtual object behavior. We conducted a formative study with three expert VR designers to identify five representative HOI designs to support diverse user experiences. Building upon preference data from 20 participants, we develop an HOI mapping module with in-context learning. In a user study with 12 VR interaction designers, HOI mapping from HOICraft significantly reduced trial-and-error iterations compared to manual authoring. Finally, we assessed the usability of HOICraft, demonstrating its effectiveness for HOI design in VR.

Problem

• Hand-Object Interaction
• Virtual Reality
• Interaction Design
• Part-Level Interaction
• Authoring Tool

Innovation

• Vision-Language Model (VLM)
• Part-level Interaction
• In-situ Authoring
• Hand-Object Interaction (HOI)
• In-context Learning