Enhancing IMU-Based Online Handwriting Recognition via Contrastive Learning with Zero Inference Overhead

2026-02-04

AI Summary

This work addresses the challenges of memory constraints and limited accuracy in online handwriting recognition on edge devices using inertial measurement units (IMUs). To overcome these limitations, the authors propose the Error-augmented Contrastive Handwriting Recognition (ECHWR) framework, which employs a disposable auxiliary branch during training to align IMU signals with semantic text embeddings. The framework introduces a dual-objective contrastive loss comprising an intra-batch contrastive loss and a novel error-driven contrastive loss that leverages synthetically generated hard negative samples to distinguish between correct and erroneous writing patterns. Notably, this approach incurs no additional inference overhead while significantly enhancing model generalization, particularly for unseen writing styles. On the OnHW-Words500 dataset, the method reduces character error rates by 7.4% and 10.4% under writer-independent and writer-dependent settings, respectively, outperforming current state-of-the-art approaches.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Planning, Routing, and Scheduling: Activity and Plan Recognition
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Online handwriting recognition using inertial measurement units opens up handwriting on paper as input for digital devices. Doing it on edge hardware improves privacy and lowers latency, but entails memory constraints. To address this, we propose Error-enhanced Contrastive Handwriting Recognition (ECHWR), a training framework designed to improve feature representation and recognition accuracy without increasing inference costs. ECHWR utilizes a temporary auxiliary branch that aligns sensor signals with semantic text embeddings during the training phase. This alignment is maintained through a dual contrastive objective: an in-batch contrastive loss for general modality alignment and a novel error-based contrastive loss that distinguishes between correct signals and synthetic hard negatives. The auxiliary branch is discarded after training, which allows the deployed model to keep its original, efficient architecture. Evaluations on the OnHW-Words500 dataset show that ECHWR significantly outperforms state-of-the-art baselines, reducing character error rates by up to 7.4% on the writer-independent split and 10.4% on the writer-dependent split. Finally, although our ablation studies indicate that solving specific challenges require specific architectural and objective configurations, error-based contrastive loss shows its effectiveness for handling unseen writing styles.

Problem

• IMU-based handwriting recognition
• edge computing
• memory constraints
• inference overhead
• handwriting recognition accuracy

Innovation

• contrastive learning
• inertial measurement unit (IMU)
• handwriting recognition
• zero inference overhead
• error-based contrastive loss