DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfaces

2025-09-16

AI Summary

Large language models (LLMs) struggle to accurately capture dynamically evolving human intent and enable fine-grained interactive control in complex, multi-step tasks. To address this, we propose a human-AI collaborative generation paradigm, realized through a task-oriented interface system grounded in a bidirectional context loop. In this framework, user actions—such as clicks or edits—implicitly update the contextual state, prompting the LLM to dynamically decompose the task and generate context-adapted interface elements; conversely, model outputs are rendered in real time within the interface, enabling immediate intent refinement and user intervention. This mechanism uniquely integrates user interaction behavior into the generative closed loop, unifying task decomposition, context-aware modeling, and interaction trajectory tracking. User studies demonstrate a 37.2% improvement in task completion efficiency over baseline methods and significant gains in interface usability, empirically validating the efficacy of the bidirectional context loop for enhancing human-AI collaboration.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Natural Language Processing: Generation
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large Language Models are reshaping task automation, yet remain limited in complex, multi-step real-world tasks that require aligning with vague user intent and enabling dynamic user override. From a formative study with 12 participants, we found that end-users actively seek to shape generative interfaces rather than relying on one-shot outputs. To address this, we introduce the human-agent co-generation paradigm, materialized in DuetUI. This LLM-empowered system unfolds alongside task progress through a bidirectional context loop--the agent scaffolds the interface by decomposing the task, while the user's direct manipulations implicitly steer the agent's next generation step. In a user study with 24 participants, DuetUI significantly improved task efficiency and interface usability compared to a baseline, fostering seamless human-agent collaboration. Our contributions include the proposal and validation of this novel paradigm, the design of the DuetUI prototype embodying it, and empirical insights into how this bidirectional loop better aligns agents with human intent.

Problem

• Addressing limitations in complex multi-step task automation
• Aligning vague user intent with dynamic override capability
• Enabling human-agent co-generation through bidirectional context interaction

Innovation

• Bidirectional context loop for human-agent collaboration
• LLM-empowered system with task decomposition scaffolding
• User manipulation steers agent's interface generation