GovScape: A Public Multimodal Search System for 70 Million Pages of Government PDFs

2025-11-14

AI Summary

Government PDF documents suffer from poor discoverability and limited retrieval capabilities—typically supporting only download and keyword-based search. Method: This paper introduces the first open-source multimodal search system designed for a large-scale federal government PDF corpus (70.95 million pages), integrating OCR-based text extraction, metadata parsing, text/image embedding, and hybrid retrieval via Elasticsearch and vector databases. Contribution/Results: The system enables metadata filtering, exact and semantic text search, and visual content retrieval (e.g., “redacted documents” or “pie charts”), achieving the first scalable semantic–visual joint search over tens of millions of PDFs. With a preprocessing cost of just $0.000021 per page ($21 per million pages)—equivalent to processing 47,000 pages per dollar—the system demonstrates high scalability and efficiency. It provides a reusable architectural blueprint and practical paradigm for multimodal retrieval over billion-scale government document collections.

Technology Category

• Search and Optimization: Distributed Search
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Efforts over the past three decades have produced web archives containing billions of webpage snapshots and petabytes of data. The End of Term Web Archive alone contains, among other file types, millions of PDFs produced by the federal government. While preservation with web archives has been successful, significant challenges for access and discoverability remain. For example, current affordances for browsing the End of Term PDFs are limited to downloading and browsing individual PDFs, as well as performing basic keyword search across them. In this paper, we introduce GovScape, a public search system that supports multimodal searches across 10,015,993 federal government PDFs from the 2020 End of Term crawl (70,958,487 total PDF pages) - to our knowledge, all renderable PDFs in the 2020 crawl that are 50 pages or under. GovScape supports four primary forms of search over these 10 million PDFs: in addition to providing (1) filter conditions over metadata facets including domain and crawl date and (2) exact text search against the PDF text, we provide (3) semantic text search and (4) visual search against the PDFs across individual pages, enabling users to structure queries such as"redacted documents"or"pie charts."We detail the constituent components of GovScape, including the search affordances, embedding pipeline, system architecture, and open source codebase. Significantly, the total estimated compute cost for GovScape's pre-processing pipeline for 10 million PDFs was approximately $1,500, equivalent to 47,000 PDF pages per dollar spent on compute, demonstrating the potential for immediate scalability. Accordingly, we outline steps that we have already begun pursuing toward multimodal search at the 100+ million PDF scale. GovScape can be found at https://www.govscape.net.

Problem

• Limited access to millions of government PDFs in web archives
• Basic keyword search insufficient for complex document discovery
• Need multimodal search capabilities across PDF content types

Innovation

• Multimodal search system for government PDFs
• Semantic and visual search across PDF pages
• Scalable preprocessing pipeline for large datasets