Interaction-Augmented Instruction: Modeling the Synergy of Prompts and Interactions in Human-GenAI Collaboration

2025-10-29

AI Summary

Textual prompts struggle to precisely encode fine-grained, referential intentions, and existing work lacks a unified formalism for modeling the synergy between textual instructions and GUI interactions (e.g., brushing, clicking), hindering human-AI collaboration efficiency. Method: We propose the Interaction-Augmented Instruction (IAI) model—a novel framework that formally unifies instruction and interaction via an entity-relation graph. We distill 12 atomic interaction primitives that are composable, descriptive, discriminative, and generative. Leveraging iterative deductive reasoning, IAI integrates structural modeling with behavioral analysis to enable systematic abstraction of multimodal inputs and cross-tool design transfer. Contribution/Results: Case studies demonstrate that IAI effectively guides the construction, optimization, and extensibility of interactive designs—establishing the first principled, graph-based paradigm for instruction–interaction co-modeling in human-AI collaborative interfaces.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Intelligent Robots: Human-Robot Interaction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Text prompt is the most common way for human-generative AI (GenAI) communication. Though convenient, it is challenging to convey fine-grained and referential intent. One promising solution is to combine text prompts with precise GUI interactions, like brushing and clicking. However, there lacks a formal model to model synergistic designs between prompts and interactions, hindering their comparison and innovation. To fill this gap, via an iterative and deductive process, we develop the Interaction-Augmented Instruction (IAI) model, a compact entity-relation graph formalizing how the combination of interactions and text prompts enhances human-generative AI communication. With the model, we distill twelve recurring and composable atomic interaction paradigms from prior tools, verifying our model's capability to facilitate systematic design characterization and comparison. Case studies further demonstrate the model's utility in applying, refining, and extending these paradigms. These results illustrate our IAI model's descriptive, discriminative, and generative power for shaping future GenAI systems.

Problem

• Modeling synergistic designs between prompts and interactions
• Addressing fine-grained intent communication in human-GenAI collaboration
• Developing formal framework for systematic design comparison

Innovation

• Combining text prompts with precise GUI interactions
• Developing entity-relation graph model for synergistic designs
• Distilling twelve recurring atomic interaction paradigms