A Cross-Country Analysis of GDPR Cookie Banners and Flexible Methods for Scraping Them

2025-03-25

AI Summary

This study investigates the compliance and ethical challenges of online tracking consent mechanisms under the GDPR and ePrivacy Directive. Method: We systematically evaluated cookie banner practices across 254,000 web pages from the top 10,000 websites in 31 countries, employing a custom browser automation framework, a multilingual banner detection model, CMP fingerprinting, and a distributed crawler infrastructure. Contribution/Results: We find that 67% of sites deploy consent interfaces—67% managed by Consent Management Platforms (CMPs), with three vendors dominating 37% of the market. Only 15% meet minimum legal requirements (e.g., explicit “reject all” option); regulatory enforcement has not significantly improved compliance; and CMP type accounts for 18% of observed compliance variance. The study delivers the first large-scale, empirically grounded benchmark for global privacy governance, supports evidence-based regulatory intervention, and releases an open-source observatory platform (consent-observatory.eu) with fully reproducible methodology.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Application Domains: Transportation
• Machine Learning: Privacy

Application Category

• Responsible Web: Consent frameworks and practices on the web
• Security and Privacy: Data transparency and provenance
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

Online tracking remains problematic, with compliance and ethical issues persisting despite regulatory efforts. Consent interfaces, the visible manifestation of this industry, have seen significant attention over the years. We present robust automated methods to study the presence, design, and third-party suppliers of consent interfaces at scale and the web service consent-observatory.eu to do it with. We examine the top 10,000 websites across 31 countries under the ePrivacy Directive and GDPR (n=254.148). Our findings show that 67% of websites use consent interfaces, but only 15% are minimally compliant, mostly because they lack a reject option. Consent management platforms (CMPs) are powerful intermediaries in this space: 67% of interfaces are provided by CMPs, and three organisations hold 37% of the market. There is little evidence that regulators' guidance and fines have impacted compliance rates, but 18% of compliance variance is explained by CMPs. Researchers should take an infrastructural perspective on online tracking and study the factual control of intermediaries to identify effective leverage points.

Problem

• Analyzing GDPR cookie banners' compliance and design issues
• Developing automated methods to study consent interfaces at scale
• Investigating CMPs' influence on tracking compliance and market control

Innovation

• Automated scraping of GDPR cookie banners
• Large-scale analysis across 31 countries
• Focus on consent management platforms impact