Chronicling Germany: An Annotated Historical Newspaper Dataset

2024-01-30

AI Summary

Historical German newspapers exhibit dense layouts and degraded typography, severely impairing layout analysis and OCR accuracy—thus hindering NLP applications in digital humanities. To address this, we introduce the first fine-grained, jointly annotated dataset for German historical newspapers (1852–1924), comprising 693 pages annotated across five structural levels: columns, articles, paragraphs, lines, and characters. Crucially, it is the first to provide synchronized layout and text annotations. Leveraging this resource, we establish a cross-domain generalization benchmark to advance low-resource historical document visual understanding. We validate its utility on standard OCR and layout analysis pipelines, achieving high in-domain accuracy. All data and code are publicly released, significantly enhancing reproducibility and scalability in historical document digitization.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Application Domains: Humanities & Computational Social Science
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

The correct detection of dense article layout and the recognition of characters in historical newspaper pages remains a challenging requirement for Natural Language Processing (NLP) and machine learning applications on historical newspapers in the field of digital history. Digital newspaper portals for historic Germany typically provide Optical Character Recognition (OCR) text, albeit of varying quality. Unfortunately, layout information is often missing, limiting this rich source's scope. Our dataset is designed to enable the training of layout and OCR modells for historic German-language newspapers. The Chronicling Germany dataset contains 693 annotated historical newspaper pages from the time period between 1852 and 1924. The paper presents a processing pipeline and establishes baseline results on in- and out-of-domain test data using this pipeline. Both our dataset and the corresponding baseline code are freely available online. This work creates a starting point for future research in the field of digital history and historic German language newspaper processing. Furthermore, it provides the opportunity to study a low-resource task in computer vision

Problem

• Detecting dense article layout in historical newspapers
• Improving OCR accuracy for historic German-language texts
• Providing annotated dataset for digital history research

Innovation

• Annotated dataset for layout and OCR training
• Processing pipeline for historical newspaper analysis
• Baseline results on in-and out-of-domain data