Problem
Research questions and friction points this paper is trying to address.
computer-use agents
observation interface
dynamic content perception
audio-visual observation
agent perception limitations
Agent-Computer Observation Interfaces Enable Dynamic Computer Use
🤖 AI Summary
Existing agents are constrained by low-frequency screenshot observations, rendering them unable to perceive dynamic information such as video, audio, and transient UI events. This work proposes the Agent-Computer Observation Interface (AOI), which treats the observation interface as an independent design dimension for the first time. By decoupling continuous adaptive perception from discrete actions, AOI establishes a model-agnostic perceptual layer. It employs a three-component gating mechanism—keyframe capture, volume-triggered speech transcription, and visual narrative textualization—to enable efficient perception of dynamic computing environments. Evaluated on DynaCU-Bench, AOI boosts performance across diverse computer-use (CU) models by 17–48 percentage points without any retraining, elevating audio task success rates from near 0% to 100%.
📝 Abstract
SWE-agent established the action interface as an underexplored design axis for software-engineering agents; we make the analogous case for the observation interface in computer-use (CU) agents. Current CU agents, closed and open-source alike, tie observation to action--one screenshot every 3-5 s, no audio--leaving them blind and deaf between screenshots to video, animations, transient UI events, meetings, and spoken instructions. We introduce the Agent-Computer Observation Interface (AOI), a model-agnostic perception layer that decouples continuous, adaptive observation from discrete actions through three gated components: inter-step keyframe capture, volume-gated audio transcription, and CU-model-generated visual narration that persists as text. Each produces almost nothing on static, silent content, reducing to the standard loop without degrading it.
On DynaCU-Bench (100 dynamic browser tasks plus a 50-task static control), CU models from 7B to frontier scale gain +17 to +48 pp over their screenshot baselines with zero retraining, turning tasks that are near-impossible from periodic screenshots into largely solved ones. The gap is starkest on audio: on a spoken-content subset AOI agents solve every task, whereas streaming voice models hear accurately but cannot act on what they hear without the scaffold. The decomposition is as informative as the headline gain: keyframe selection turns out not to matter--the value comes from narrating captured frames into persistent text--and the interface is not a fixed bundle, since on a newer model (Gemini 3 Flash) the keyframe stream actively regresses through image-token dilution, so its components must be selected per model rather than shipped as one configuration.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Agent-Computer Observation Interface
dynamic computer use
model-agnostic perception
persistent visual narration
audio-aware agents
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
