Problem
Research questions and friction points this paper is trying to address.
web automation
interaction patterns
programmable workflows
end-user programming
automation discovery
Motif: Discovering and Automating Personal Web Workflows
🤖 AI Summary
Users often fail to recognize automation opportunities in routine web interactions, and existing large language model–based approaches are costly and require explicit task specification. This work proposes an unobtrusive automation framework that passively monitors browser activity to automatically identify programmable, repetitive interaction patterns and generates installable automation scripts, which users can review and refine using natural language. The approach introduces the first mechanism for discovering automation opportunities without active user involvement, integrating behavioral tracking, pattern recognition, program synthesis, and a large language model interface. A user study demonstrates that the system uncovers substantially more automation opportunities than users identify on their own; most discovered automations align with real-world workflows and are rated as useful, yielding high user willingness to adopt.
📝 Abstract
Recent advances in LLMs and existing work on programming by demonstration have made it possible for end users to create automations by explicitly demonstrating their behavior to LLMs. However, these approaches rely on the assumption that users know what to automate and what is capable of being automated. Additionally, automation via LLM agents is often expensive compared with programs. We introduce Motif, a system that passively observes everyday browser activity to discover recurring interaction patterns that are programmable, makes recommendations to users whenever a pattern is discovered and generate a program to install after user confirmation. Users can review, and refine the program using natural language. We evaluated Motif in a multi-day study, comparing its ambient discoveries against automations users attempted to build via ``vibe coding.'' With eight participants, Motif discovered more automatable patterns than users recognized. Most of them matched participants' routines and were useful. Follow-up surveys showed most would continue using Motif-generated programs.
Innovation
Methods, ideas, or system contributions that make the work stand out.
passive observation
workflow discovery
programmable interaction patterns
natural language refinement
browser automation
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
