Overhearing LLM Agents: A Survey, Taxonomy, and Roadmap

2025-09-19

AI Summary

Traditional LLM-based agents rely heavily on explicit user commands, leading to frequent interruptions and reduced usability. Method: This paper introduces the “Overhearing Agents” paradigm—a novel approach where agents passively monitor multimodal environmental interactions (e.g., speech, screen activity) and intervene non-intrusively only when contextually appropriate. Grounded in HCI research and LLM agent surveys, we establish the first task taxonomy for overhearing, categorizing intervention opportunities into three types—predictive, responsive, and corrective—each accompanied by design principles. We further propose a systematic architectural framework and practical developer guidelines. Contribution/Results: We formally define the “listening-first” human–AI interaction paradigm, present the first theoretical classification framework for overhearing tasks, and empirically validate its feasibility and low-interruption advantage across healthcare, education, and office settings—laying foundational groundwork for silent, context-aware collaborative AI systems.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Modeling other Agents
• Humans and AI: Interaction Techniques and Devices

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions

Abstract

Imagine AI assistants that enhance conversations without interrupting them: quietly providing relevant information during a medical consultation, seamlessly preparing materials as teachers discuss lesson plans, or unobtrusively scheduling meetings as colleagues debate calendars. While modern conversational LLM agents directly assist human users with tasks through a chat interface, we study this alternative paradigm for interacting with LLM agents, which we call "overhearing agents." Rather than demanding the user's attention, overhearing agents continuously monitor ambient activity and intervene only when they can provide contextual assistance. In this paper, we present the first analysis of overhearing LLM agents as a distinct paradigm in human-AI interaction and establish a taxonomy of overhearing agent interactions and tasks grounded in a survey of works on prior LLM-powered agents and exploratory HCI studies. Based on this taxonomy, we create a list of best practices for researchers and developers building overhearing agent systems. Finally, we outline the remaining research gaps and reveal opportunities for future research in the overhearing paradigm.

Problem

• Studying LLM agents that monitor ambient activity unobtrusively
• Establishing taxonomy for overhearing agent interactions and tasks
• Identifying research gaps and opportunities for future development

Innovation

• Overhearing agents monitor ambient activity continuously
• Intervene only when providing contextual assistance
• Establish taxonomy and best practices for development