From Historical Tabular Image to Knowledge Graphs: A Provenance-Aware Modular Pipeline

2026-05-06

AI Summary

This work addresses the challenges of converting handwritten historical tabular images into structured knowledge graphs, which are hindered by multimodal processing complexity and the opacity of end-to-end approaches that impede human oversight and trustworthy evaluation. The authors propose a modular, traceable three-stage pipeline that sequentially performs table reconstruction, information extraction, and knowledge graph construction, integrating systematic data provenance throughout to ensure every entity and literal can be traced back to its original visual or textual source. This design significantly enhances transparency and controllability in human–machine collaboration. Experiments on real-world military service records demonstrate that the modular architecture, combined with human intervention, effectively handles the intricacies of historical data transformation, while three variants of table reconstruction underscore the critical role of process transparency in ensuring output quality.

Abstract

Handwritten archival tables contain rich historical information, yet transforming them into structured representations, such as Knowledge Graphs, requires integrating table structure recognition, handwriting recognition, and semantic interpretation - a complex multimodal process. End-to-end AI implementations can obscure these steps, resulting in opaque algorithmic operations that hinder human oversight, critical assessment, and trust. To address this, we present a modular, provenance-aware pipeline to convert handwritten tabular images into KGs supporting human-AI collaboration. The pipeline decomposes the workflow into three stages - table reconstruction, information extraction, and KG construction - while exposing intermediate representations for inspection, evaluation, and correction. A key contribution of our approach is the systematic integration of data provenance at every stage, ensuring that all extracted entities and literals remain traceable to their visual and textual origins. The proposed pipeline is demonstrated through a number of experiments on real-world archival material concerning military careers. The results across three different table reconstruction variants highlight the importance of modularisation. By coupling modularity with data provenance, our work advances transparent and collaboratively controllable image-to-KG pipelines for complex historical data.

Problem

• handwritten tabular images
• knowledge graphs
• data provenance
• human-AI collaboration
• historical archives

Innovation

• modular pipeline
• data provenance
• knowledge graph construction
• handwritten table recognition
• human-AI collaboration