Problem
Research questions and friction points this paper is trying to address.
span-level masking
position constraint
language model pre-training
POSPAN: Position-Constrained Span Masking for Language Model Pre-training
🤖 AI Summary
本文提出POSPAN框架，通过结合跨度长度分布与位置约束分布来改进语言模型预训练中的跨度掩码策略，增强了模型性能。
📝 Abstract
Span-level masked language modeling (MLM) has shown to be advantageous to pre-trained language models over the original single-token MLM, as entities/phrases and their dependencies are critical to language understanding. Previous works only consider span length with some discrete distributions, while the dependencies among spans are ignored, i.e., assuming that the positions of masked spans are uniformly distributed. In this paper, we present POSPAN, a general framework to allow diverse position-constrained span masking strategies via the combination of span length distribution and position constraint distribution, which unifies all existing span-level masking methods. To verify the effectiveness of POSPAN in pre-training, we evaluate it on the datasets from several NLU benchmarks. Experimental results indicate that the position constraint is capable of enhancing span-level masking broadly, and our best POSPAN setting consistently outperforms its span-length-only counterparts and vanilla MLM. We also conduct theoretical analysis for the position constraint in masked language models to shed light on the reason why POSPAN works well, demonstrating the rationality and necessity of POSPAN.
Innovation
Methods, ideas, or system contributions that make the work stand out.
position-constrained
span masking
language model pre-training
🔎 Similar Papers
No similar papers found.
