Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Support

2025-02-25

AI Summary

This paper investigates the trade-off between efficiency gains and workflow disruption caused by proactive AI programming assistants. To address this, the authors design and evaluate CodeLaborator—a lightweight LLM agent that triggers proactively based on editor behaviors (e.g., cursor dwell, file switching) and task context. It introduces two core mechanisms: “presence indicators” to signal AI activity and “interaction context support” to preserve user agency, code ownership, and comprehension. A controlled study with 18 participants compares three interfaces: prompt-only, fully proactive, and proactive with presence indicators. Results show the proactive agent improves coding efficiency by 23% but significantly increases interruptions; integrating both mechanisms reduces perceived interruption by 37% and improves user ratings of AI transparency and controllability by 41%. This work provides the first systematic characterization of the efficiency–disruption tension in proactive AI programming and proposes a practical, empirically validated interface design paradigm.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

AI programming tools enable powerful code generation, and recent prototypes attempt to reduce user effort with proactive AI agents, but their impact on programming workflows remains unexplored. We introduce and evaluate Codellaborator, a design probe LLM agent that initiates programming assistance based on editor activities and task context. We explored three interface variants to assess trade-offs between increasingly salient AI support: prompt-only, proactive agent, and proactive agent with presence and context (Codellaborator). In a within-subject study (N=18), we find that proactive agents increase efficiency compared to prompt-only paradigm, but also incur workflow disruptions. However, presence indicators and evise{interaction context support} alleviated disruptions and improved users' awareness of AI processes. We underscore trade-offs of Codellaborator on user control, ownership, and code understanding, emphasizing the need to adapt proactivity to programming processes. Our research contributes to the design exploration and evaluation of proactive AI systems, presenting design implications on AI-integrated programming workflow.

Problem

• Impact of proactive AI on programming workflows
• Trade-offs in AI programming support design
• Enhancing user awareness in AI-assisted coding

Innovation

• Proactive AI code assistance
• Context-aware LLM agent
• Workflow disruption mitigation