Dependency Parsing is More Parameter-Efficient with Normalization

2025-05-26

AI Summary

In dependency parsing, the biaffine scoring mechanism suffers from parameter redundancy, inefficient training, and poor generalization due to the absence of explicit normalization. This paper provides the first theoretical and empirical demonstration that explicit normalization of biaffine scores prior to softmax is essential. To address this, we propose a normalized biaffine scoring module incorporating input variance correction, integrated into a single-hop parser based on an N-layer BiLSTM architecture. Evaluated on six mainstream dependency treebanks, our approach achieves state-of-the-art performance on two datasets. Moreover, it significantly reduces model parameters, decreases training sample requirements, and simultaneously improves robustness and generalization. The proposed method establishes a new paradigm for efficient, lightweight dependency parsing.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Cognitive Modeling & Cognitive Systems: Affective Computing
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Dependency parsing is the task of inferring natural language structure, often approached by modeling word interactions via attention through biaffine scoring. This mechanism works like self-attention in Transformers, where scores are calculated for every pair of words in a sentence. However, unlike Transformer attention, biaffine scoring does not use normalization prior to taking the softmax of the scores. In this paper, we provide theoretical evidence and empirical results revealing that a lack of normalization necessarily results in overparameterized parser models, where the extra parameters compensate for the sharp softmax outputs produced by high variance inputs to the biaffine scoring function. We argue that biaffine scoring can be made substantially more efficient by performing score normalization. We conduct experiments on six datasets for semantic and syntactic dependency parsing using a one-hop parser. We train N-layer stacked BiLSTMs and evaluate the parser's performance with and without normalizing biaffine scores. Normalizing allows us to beat the state of the art on two datasets, with fewer samples and trainable parameters. Code: https://anonymous.4open.science/r/EfficientSDP-70C1

Problem

• Lack of normalization in biaffine scoring causes overparameterized parser models
• Biaffine scoring efficiency can be improved by score normalization
• Normalization enhances dependency parsing performance with fewer parameters

Innovation

• Normalization improves biaffine scoring efficiency
• Fewer parameters achieve state-of-the-art performance
• BiLSTMs validate normalization benefits empirically