Not Here, Go There: Analyzing Redirection Patterns on the Web

2025-07-29

AI Summary

URI redirection is critical for SEO, website migration, and web security, yet inconsistent practices impair usability, degrade search rankings, and threaten digital preservation. To address this, we construct a large-scale redirection chain tracing system—covering 11 million unique URIs and supporting up to 10 hops—integrating HTTP status code analysis, link resolution, and soft-404 detection. Our study is the first to empirically uncover three pervasive issues at scale: (1) 50% of redirection chains terminate in failure; (2) 0.06% exceed 10 hops, forming “redirection storms”; and (3) we identify 62,000 custom 404 pages, 48% of which exhibit misleading soft-404 behavior. Furthermore, non-canonical redirections and “URI convergence” patterns correlate strongly with brand transitions, server instability, and potential open redirection vulnerabilities. These findings provide empirical foundations and methodological tools for improving redirection governance, enhancing SEO resilience, and advancing web sustainability.

Technology Category

• Reasoning under Uncertainty: Causality
• Search and Optimization: Other Foundations of Search & Optimization
• Data Mining & Knowledge Management: Web

Application Category

• Responsible Web: Sustainability and climate impact of web technologies
• Web Mining and Content Analysis: Bridging structured and unstructured data
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

URI redirections are integral to web management, supporting structural changes, SEO optimization, and security. However, their complexities affect usability, SEO performance, and digital preservation. This study analyzed 11 million unique redirecting URIs, following redirections up to 10 hops per URI, to uncover patterns and implications of redirection practices. Our findings revealed that 50% of the URIs terminated successfully, while 50% resulted in errors, including 0.06% exceeding 10 hops. Canonical redirects, such as HTTP to HTTPS transitions, were prevalent, reflecting adherence to SEO best practices. Non-canonical redirects, often involving domain or path changes, highlighted significant web migrations, rebranding, and security risks. Notable patterns included "sink" URIs, where multiple redirects converged, ranging from traffic consolidation by global websites to deliberate "Rickrolling." The study also identified 62,000 custom 404 URIs, almost half being soft 404s, which could compromise SEO and user experience. These findings underscore the critical role of URI redirects in shaping the web while exposing challenges such as outdated URIs, server instability, and improper error handling. This research offers a detailed analysis of URI redirection practices, providing insights into their prevalence, types, and outcomes. By examining a large dataset, we highlight inefficiencies in redirection chains and examine patterns such as the use of "sink" URIs and custom error pages. This information can help webmasters, researchers, and digital archivists improve web usability, optimize resource allocation, and safeguard valuable online content.

Problem

• Analyzing URI redirection patterns and their impact on web usability
• Investigating SEO and security implications of canonical and non-canonical redirects
• Identifying inefficiencies in redirection chains and error handling practices

Innovation

• Analyzed 11M URIs with 10-hop redirections
• Identified canonical and non-canonical redirect patterns
• Detected sink URIs and custom 404 issues