Learning to Align: Addressing Character Frequency Distribution Shifts in Handwritten Text Recognition

2025-06-11

AI Summary

In handwritten text recognition, historical and geographical variations induce shifts in character frequency distributions, degrading the generalization of generic models on domain-specific subsets. To address this, we propose a Wasserstein-distance-based distribution alignment framework: (1) we explicitly formulate character frequency distribution alignment as a differentiable loss term for the first time; (2) we design a distribution-aware training objective and enable inference-time, distribution-guided beam search decoding—without requiring model retraining. Our method thus jointly optimizes training performance and supports plug-and-play deployment. Extensive experiments across multiple handwritten datasets and diverse recognition architectures demonstrate consistent and substantial improvements in overall accuracy and robustness—particularly on distribution-shifted subsets. The source code is publicly available.

Technology Category

• Search and Optimization: Distributed Search
• Machine Learning: Distributed Machine Learning & Federated Learning
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Handwritten text recognition aims to convert visual input into machine-readable text, and it remains challenging due to the evolving and context-dependent nature of handwriting. Character sets change over time, and character frequency distributions shift across historical periods or regions, often causing models trained on broad, heterogeneous corpora to underperform on specific subsets. To tackle this, we propose a novel loss function that incorporates the Wasserstein distance between the character frequency distribution of the predicted text and a target distribution empirically derived from training data. By penalizing divergence from expected distributions, our approach enhances both accuracy and robustness under temporal and contextual intra-dataset shifts. Furthermore, we demonstrate that character distribution alignment can also improve existing models at inference time without requiring retraining by integrating it as a scoring function in a guided decoding scheme. Experimental results across multiple datasets and architectures confirm the effectiveness of our method in boosting generalization and performance. We open source our code at https://github.com/pkaliosis/fada.

Problem

• Addressing character frequency shifts in handwritten text recognition
• Improving model accuracy under temporal and contextual dataset shifts
• Enhancing existing models without retraining via guided decoding

Innovation

• Novel Wasserstein distance-based loss function
• Guided decoding with distribution alignment
• Improves models without retraining