AI-Instruments: Embodying Prompts as Instruments to Abstract&Reflect Graphical Interface Commands as General-Purpose Tools

2025-02-26

AI Summary

In AI-assisted design, linear, chat-based prompting impedes exploration of ambiguous intentions, backtracking revisions, and directional control. This paper proposes the “intention embodiment” framework: transforming natural language prompts into reusable, directly manipulable interface tools that support multiple interpretations of user intent and enable real-time dual reflection—on both the intent layer (user’s evolving goals) and the response layer (AI’s generated outputs). Our method integrates LLM-driven prompt understanding and tool auto-generation, polymorphic intention modeling, and the technical probe methodology, instantiated in four image-generation scenarios. A user study with 12 participants demonstrates significant improvements in intent expression accuracy, nonlinear iterative efficiency, and directional controllability in human-AI co-creation. The core contributions are the first formalization of intention embodiment as a design paradigm and the introduction of dual reflection—a novel interaction mechanism that transcends the limitations of conventional text-only interfaces.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Intelligent Robots: Embodied AI
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Chat-based prompts respond with verbose linear-sequential texts, making it difficult to explore and refine ambiguous intents, back up and reinterpret, or shift directions in creative AI-assisted design work. AI-Instruments instead embody"prompts"as interface objects via three key principles: (1) Reification of user-intent as reusable direct-manipulation instruments; (2) Reflection of multiple interpretations of ambiguous user-intents (Reflection-in-intent) as well as the range of AI-model responses (Reflection-in-response) to inform design"moves"towards a desired result; and (3) Grounding to instantiate an instrument from an example, result, or extrapolation directly from another instrument. Further, AI-Instruments leverage LLM's to suggest, vary, and refine new instruments, enabling a system that goes beyond hard-coded functionality by generating its own instrumental controls from content. We demonstrate four technology probes, applied to image generation, and qualitative insights from twelve participants, showing how AI-Instruments address challenges of intent formulation, steering via direct manipulation, and non-linear iterative workflows to reflect and resolve ambiguous intents.

Problem

• Enhancing ambiguous intent exploration in AI design
• Transforming prompts into reusable interface objects
• Facilitating non-linear iterative workflows via AI-Instruments

Innovation

• Reifies user-intent as reusable instruments
• Reflects multiple ambiguous intents interpretations
• Grounds instruments from examples or results