Stringalign: Moving beyond summary statistics with a transparent Unicode-aware tool for evaluating automatic transcription models

2026-06-14

AI Summary

This work addresses the inconsistency and poor reproducibility of character error rate (CER) and word error rate (WER) metrics in evaluating automatic transcription models, which stem from opaque preprocessing and ambiguous definitions of characters and words in existing text alignment tools. To resolve this, the authors propose Stringalign, a lightweight Python library that introduces FAIR principles to transcription evaluation for the first time. Stringalign enables reproducible, fine-grained error analysis through Unicode-aware transparent normalization, flexible tokenization strategies, and character- and word-level alignment algorithms. Coupled with interactive visualizations, Stringalign significantly enhances evaluation consistency and interpretability across OCR, handwritten text recognition (HTR), and automatic speech recognition (ASR) tasks, thereby facilitating effective model diagnosis and selection.

Abstract

Comparing text strings is crucial when evaluating and understanding the performance of various text processing tasks such as document recognition and audio transcription. With an increasingly complex landscape of AI-based handwritten text recognition (HTR), optical character recognition (OCR) and automatic speech recognition (ASR) models, there is a need for tools that facilitate evaluation in a flexible and reproducible way. This paper presents Stringalign, a Python library designed to simplify the evaluation process for automatic transcription projects and facilitate transparent evaluation. Stringalign's tools to examine and visualise both the rate of errors and the types of errors a model makes, give insights into possible improvements and help inform model selection for a particular task. Widely used string comparison metrics, such as the character and word error rates (CER and WER), although useful, can be ambiguous due to varying definitions of what constitutes a character and a word. Stringalign addresses this challenge by ensuring all preprocessing (i.e. normalisation and tokenisation) is transparent and easily replicable, and by providing tools to move beyond summary statistics and analyse common model errors. Moreover, Stringalign adheres to FAIR (Findable, Accessible, Interoperable, and Reusable) principles for research software while staying lightweight and easy to adapt into researchers existing workflows. In this paper, we discuss challenges with character and word level string comparisons and show through examples that where existing tools can yield opaque and sometimes confusing results, Stringalign provides an easy-to-use and unambiguous alternative.

Problem

• automatic transcription evaluation
• string comparison
• character error rate
• word error rate
• evaluation transparency

Innovation

• Stringalign
• Unicode-aware evaluation
• transparent preprocessing
• error analysis
• FAIR software