Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces

2026-05-14

AI Summary

This work reveals a previously unexamined vulnerability in large language model (LLM) browser agents: their underlying models can be passively fingerprinted through UI interaction patterns during web tasks, exposing them to targeted attacks. The study introduces the first systematic analysis of this risk and proposes a JavaScript-based passive tracking method that leverages behavioral sequence modeling and machine learning classifiers to identify the specific LLM powering an agent with high accuracy from minimal early interactions. Evaluated across 14 prominent LLMs and four distinct web environments, the approach achieves up to 96% F1 score, demonstrating strong generalization across model scales and families. To facilitate further research, the authors publicly release the collected interaction trajectory dataset and the evaluation framework.

Abstract

As LLM-based agents increasingly browse the web on users' behalf, a natural question arises: can websites passively identify which underlying model powers an agent? Doing so would represent a significant security risk, enabling targeted attacks tailored to known model vulnerabilities. Across 14 frontier LLMs and four web environments spanning information retrieval and shopping tasks, we show that an agent's actions and interaction timings, captured via a passive JavaScript tracker, are sufficient to identify the underlying model with up to 96\% F1. We formalise this attack surface by demonstrating that classifiers trained on agent actions generalise across model sizes and families. We further show that strong classifiers can be trained from few interaction traces and that agent identity can be inferred early within an episode. Injecting randomised timing delays between actions substantially degrades classifier performance, but does not provide robust protection: a classifier retrained on delayed traces largely recovers performance. We release our harness and a labelled corpus of agent traces \href{https://github.com/KabakaWilliam/known_actions}{here}.

Problem

• LLM browser agents
• model fingerprinting
• UI traces
• passive identification
• web security

Innovation

• LLM fingerprinting
• browser agent identification
• UI trace analysis
• passive side-channel attack
• model inference