Beyond Chat and Clicks: GUI Agents for In-Situ Assistance via Live Interface Transformation

2026-04-16

AI Summary

This work addresses the high learning cost of complex graphical user interfaces and the limitations of existing assistance methods, which often rely on separate chat windows or require extensive custom development. The authors propose an in-situ assistance paradigm that leverages a browser extension to perform lightweight, reversible interventions on the DOM, enabling dynamic interface restructuring without altering the underlying application logic. They introduce the first DOM-based design space and computational pipeline for in-situ assistance, integrating natural language understanding, UI element localization, and reversible operations to support real-time injection of hints, highlighting of controls, or layout rearrangements on arbitrary web pages. Evaluations on two complex interfaces demonstrate the approach’s efficacy and reliability, with user studies showing significant improvements over the ChatGPTAtlas baseline in both usability and task completion efficiency.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Application Domains: Humanities & Computational Social Science
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Complex visual interfaces are powerful yet have a steep learning curve, as users must navigate feature-rich visual interfaces while reasoning about domain-specific operations. Existing approaches either deliver assistance through a separate chat-based interaction, or require substantial application-specific engineering to build support natively into each interface. To address the gaps, we propose in-situ assistance: a mode of support delivered directly within any live web interface through lightweight, browser-level interventions on the Document Object Model (DOM), without rebuilding the application or modifying its underlying logic. We contribute a design space and a computational pipeline for DOM-mediated in-situ assistance, characterizing how GUI agents can insert, mutate, or recompose web elements to make the interface easier for users to understand, use, and navigate. We instantiate in-situ assistance in DOMSteer, a Chrome extension that interprets a user's help request and live interface context, grounds it to relevant UI elements, and executes reversible DOM manipulations directly on the live page to deliver assistance, including contextual tooltips, control highlighting, layout reorganization. Quantitative evaluations on two complex visual interfaces show that DOMSteer delivers reliable and efficient in-situ assistance. Use cases and a comparative user study with baseline ChatGPTAtlas demonstrate the usability and effectiveness of DOMSteer. Altogether, these findings point to a broader role for GUI agents: not just assisting from the sidelines, but actively reconfiguring live interfaces to support users in the moment.

Problem

• GUI agents
• in-situ assistance
• visual interfaces
• user assistance
• DOM manipulation

Innovation

• in-situ assistance
• GUI agents
• DOM manipulation
• live interface transformation
• browser-level intervention