🤖 AI Summary
Scientific formulas inherently encode both syntactic structure and semantic meaning, yet these two aspects exhibit significant misalignment in their native representation spaces, limiting the performance of cross-modal retrieval. This work is the first to systematically uncover the weakly observable correspondence between formula syntax and semantics and proposes an explicit alignment approach. Specifically, it employs a graph neural network to encode syntactic structures and a text encoder to model semantic content, integrating them into a unified representation space through contrastive learning. The resulting aligned representations effectively bridge the modality gap and substantially enhance cross-modal retrieval performance, thereby demonstrating the critical role of explicit representation learning in mathematical formula understanding.
📝 Abstract
Scientific formulae are a fundamental component of scholarly communication, yet their dual nature -- as structured syntax and carriers of semantics -- remains underexplored in scholarly information retrieval. Although prior studies show that jointly modeling syntactic and semantic modalities improves retrieval performance, the relationship between their underlying representations has not been systematically investigated. In this work, we empirically study cross-modal correspondence between formula syntax and semantics. We find that their native representation spaces exhibit extremely weak observable correspondence despite strong latent correlation, indicating a substantial representation mismatch between the two modalities. We further evaluate whether this mismatch can be reduced using standard representation learning and alignment techniques. We represent syntactic structure using graph-based encoders and semantic information using text-based encoders, then apply contrastive learning to induce a shared representation space. Results show that the learned alignment substantially improves cross-modal retrieval, suggesting that explicit representation learning can recover correspondence absent from the original representation spaces.