DocTTT: Test-Time Training for Handwritten Document Recognition Using Meta-Auxiliary Learning

2025-01-22

AI Summary

Handwritten document recognition suffers from poor generalization under complex backgrounds, diverse handwriting styles, and layout variations, compounded by severe scarcity of labeled data. To address this, we propose a test-time adaptation framework that jointly integrates meta-learning (MAML-style) with masked autoencoding (MAE) into an end-to-end optimized test-time training (TTT) mechanism. Crucially, our method enables real-time visual representation adaptation from a single unlabeled test sample—eliminating reliance on large-scale annotated datasets or conventional fine-tuning. During inference, it dynamically updates model parameters, substantially enhancing robustness and generalization in few-shot settings. Evaluated on multiple standard handwritten document benchmarks, our approach surpasses current state-of-the-art methods, achieving absolute accuracy gains of 5.2–8.7 percentage points in few-shot recognition.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Despite recent significant advancements in Handwritten Document Recognition (HDR), the efficient and accurate recognition of text against complex backgrounds, diverse handwriting styles, and varying document layouts remains a practical challenge. Moreover, this issue is seldom addressed in academic research, particularly in scenarios with minimal annotated data available. In this paper, we introduce the DocTTT framework to address these challenges. The key innovation of our approach is that it uses test-time training to adapt the model to each specific input during testing. We propose a novel Meta-Auxiliary learning approach that combines Meta-learning and self-supervised Masked Autoencoder~(MAE). During testing, we adapt the visual representation parameters using a self-supervised MAE loss. During training, we learn the model parameters using a meta-learning framework, so that the model parameters are learned to adapt to a new input effectively. Experimental results show that our proposed method significantly outperforms existing state-of-the-art approaches on benchmark datasets.

Problem

• Handwritten Document Recognition
• Complex Backgrounds
• Limited Data

Innovation

• Meta-Learning
• Self-Supervised Masked Autoencoder
• Personalized Adaptation