Link prediction Graph Neural Networks for structure recognition of Handwritten Mathematical Expressions

2025-11-04

AI Summary

This paper addresses the structural recognition of handwritten mathematical expressions (HMEs), i.e., accurately modeling the 2D spatial relationships among symbols and generating symbol-label graphs. The proposed end-to-end method integrates syntactic priors with graph learning. First, a bidirectional LSTM (BLSTM) performs symbol segmentation, recognition, and coarse-grained relation classification to construct an initial graph. Second, a 2D context-free grammar (2D-CFG) parser generates syntactically valid spatial relationship constraints. Third, a graph neural network (GNN)-based link prediction module refines the graph structure under syntactic guidance. Crucially, this work is the first to embed 2D-CFG syntactic priors directly into a GNN-based link prediction framework, effectively suppressing spurious connections. Experiments on multiple benchmark datasets demonstrate significant improvements in both accuracy and robustness of structural recognition.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Natural Language Processing: Syntax — Tagging, Chunking & Parsing
• Reasoning under Uncertainty: Graphical Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Bridging structured and unstructured data

Abstract

We propose a Graph Neural Network (GNN)-based approach for Handwritten Mathematical Expression (HME) recognition by modeling HMEs as graphs, where nodes represent symbols and edges capture spatial dependencies. A deep BLSTM network is used for symbol segmentation, recognition, and spatial relation classification, forming an initial primitive graph. A 2D-CFG parser then generates all possible spatial relations, while the GNN-based link prediction model refines the structure by removing unnecessary connections, ultimately forming the Symbol Label Graph. Experimental results demonstrate the effectiveness of our approach, showing promising performance in HME structure recognition.

Problem

• Recognizing handwritten mathematical expressions using graph neural networks
• Refining symbol relationships through link prediction models
• Improving structure recognition accuracy by removing unnecessary connections

Innovation

• GNN-based link prediction for graph structure refinement
• Deep BLSTM network for symbol segmentation and recognition
• 2D-CFG parser generating possible spatial relations