Rethinking the Pointer Loss in Table Structure Recognition: Geometry-Aware Pointer Loss for Spatial Locality

2026-06-17

AI Summary

This work addresses the challenge in table structure recognition where pointer networks often mispredict spatially adjacent cells, and standard cross-entropy loss fails to account for spatial locality by treating all negative samples equally. To mitigate this, the authors propose a Geometry-Aware Pointer (GAP) loss that introduces geometric inductive bias directly into the loss function. GAP employs an inverse-distance weighting mechanism based on spatial proximity, assigning higher gradient weights to cells near the ground-truth label. This approach requires no modification to the model architecture or additional inference overhead, yet significantly reduces errors involving neighboring cells. Experimental results on PubTabNet and SynthTabNet demonstrate that GAP loss consistently improves performance and achieves state-of-the-art results.

Abstract

Table Structure Recognition (TSR) using a pointer network achieves impressive results by predicting HTML sequences while aligning tags to detected text (or cell) regions. However, our analysis reveals that when pointer networks fail, 79.6% of errors occur between spatially adjacent cells (Manhattan distance <= 2). Despite this, standard cross-entropy loss weights all negative candidates equally. In this work, we propose Geometry-Aware Pointer (GAP) Loss, which reweights the cross-entropy objective based on spatial proximity to ground truth. By applying inverse distance weighting, GAP focuses gradient flow where the model struggles most: immediate neighbors receive stronger gradients than distant cells. Our approach requires only a straightforward modification to the loss computation, maintaining the same model architecture with zero additional inference cost. Extensive experiments on PubTabNet and SynthTabNet demonstrate that GAP consistently reduces adjacent-cell errors, achieving new state-of-the-art performance. Our findings suggest that incorporating geometric inductive biases at the loss level provides a simple yet effective approach to robust TSR. Our code is available at https://github.com/teamreboott/GAP

Problem

• Table Structure Recognition
• Pointer Network
• Spatial Locality
• Cross-Entropy Loss
• Adjacent-cell Errors

Innovation

• Geometry-Aware Pointer Loss
• Table Structure Recognition
• Spatial Locality
• Pointer Network
• Inverse Distance Weighting