Letter Lemmatization: One-to-one and Banded RNNs for Reversing Character-Set Simplification and Abbreviation in Medieval Text

2026-07-10

AI Summary

This study addresses character set inconsistencies and abbreviation-related challenges in medieval texts arising from divergent scribal practices and digitization strategies. The authors propose a self-supervised character-level one-to-one RNN mapping approach, extended into a banded RNN architecture to restore abbreviations without explicit insertion or deletion operations. Innovatively integrating an “alphabetic tokenization” framework with a character semantic similarity metric, the method leverages parallel corpus alignment and heuristic algorithms to enable robust text normalization. Implemented as an efficient Python library, the approach demonstrates that as few as 20 lines of training text can reduce character error rate (CER) by 50%, substantially improving post-correction performance for handwritten text recognition. The system has been successfully applied to the expansion of abbreviations in medieval charters.

Abstract

Medieval document transcribers have very different practices; on top of that, heterogeneous digitization policies have resulted in corpora where the character-set must be viewed as fluid. In this paper we address the problem of changing between character-sets in a flexible manner. We focus on one-to-one character mappings and train characterlevel one-to-one RNNs to undo them with self-supervision; recovering half the CER even with 20 text lines. We analyse the use of these one-to-one networks for HTR post-correction and we see that they obtain significant improvements while totally ignoring ins-dels. We then use the exact same networks with character-level alignment groundtruth compiled from parallel corpora in a training and inference mode we call Banded RNNs. We use such networks to successfully expand abbreviations in medieval charter transcriptions. Finally we introduce an elaborate heuristic which takes the characters of two arbitrary character-sets and defines a metric encapsulating what we consider to be semantic similarity of characters. We call the construction of such mappings letter lemmatization and present a rich Python library that efficiently performs all presented methods.

Problem

• letter lemmatization
• character-set conversion
• medieval text
• abbreviation expansion
• text normalization

Innovation

• Letter Lemmatization
• One-to-one RNNs
• Banded RNNs
• Character-set normalization
• Medieval text processing