FP-Inconsistent: Detecting Evasive Bots using Browser Fingerprint Inconsistencies

2024-06-11

AI Summary

Evasive bots circumvent detection by manipulating browser fingerprints, rendering existing anti-bot services ineffective. This paper identifies, for the first time, the widespread phenomenon of cross-attribute and cross-temporal fingerprint inconsistency exhibited by such bots. Method: Leveraging large-scale honeypot experiments, we propose a data-driven, spatio-temporal dual-dimension approach to automatically discover inconsistency rules. Our method integrates fingerprint collection and comparison, statistical anomaly detection, and rule mining to construct a lightweight, production-deployable inconsistency detection framework compatible with commercial anti-bot services (e.g., DataDome, BotD). Contribution/Results: The framework directly integrates into existing services without requiring model retraining or infrastructure changes. A/B testing shows it reduces bot evasion rates by 48.11% on DataDome and 44.95% on BotD, empirically validating fingerprint inconsistency as a novel, robust, and practical detection signal for evasive bot traffic.

Technology Category

• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

As browser fingerprinting is increasingly being used for bot detection, bots have started altering their fingerprints for evasion. We conduct the first large-scale evaluation of evasive bots to investigate whether and how altering fingerprints helps bots evade detection. To systematically investigate evasive bots, we deploy a honey site incorporating two anti-bot services (DataDome and BotD) and solicit bot traffic from 20 different bot services that purport to sell"realistic and undetectable traffic". Across half a million requests from 20 different bot services on our honey site, we find an average evasion rate of 52.93% against DataDome and 44.56% evasion rate against BotD. Our comparison of fingerprint attributes from bot services that evade each anti-bot service individually as well as bot services that evade both shows that bot services indeed alter different browser fingerprint attributes for evasion. Further, our analysis reveals the presence of inconsistent fingerprint attributes in evasive bots. Given evasive bots seem to have difficulty in ensuring consistency in their fingerprint attributes, we propose a data-driven approach to discover rules to detect such inconsistencies across space (two attributes in a given browser fingerprint) and time (a single attribute at two different points in time). These rules, which can be readily deployed by anti-bot services, reduce the evasion rate of evasive bots against DataDome and BotD by 48.11% and 44.95% respectively.

Problem

• Robots Detection
• Browser Fingerprinting
• Evasion Techniques

Innovation

• Data-Driven Approach
• Fingerprint Inconsistency Detection
• Evasive Bots Mitigation