Extracting Problem and Method Sentence from Scientific Papers: A Context-enhanced Transformer Using Formulaic Expression Desensitization

2026-06-24

AI Summary

This study addresses the challenge that limited annotated data causes models to over-rely on specific phrasings, thereby constraining their generalization ability and impairing accurate extraction of problem and method sentences from scientific papers. To mitigate this form dependency, the authors propose a data augmentation strategy based on formalized expression desensitization and introduce a context-enhanced Transformer architecture. This architecture incorporates a context-aware mechanism to assess token importance and suppress noise. Experimental results demonstrate that the proposed approach achieves macro-F1 improvements of 3.71% and 2.67% on two scientific paper datasets, significantly outperforming baseline methods. Additionally, the study reveals that in-context learning (ICL) with large language models yields suboptimal performance on this task.

Abstract

Billions of scientific papers lead to the need to identify essential parts from the massive text. Scientific research is an activity from putting forward problems to using methods. To learn the main idea from scientific papers, we focus on extracting problem and method sentences. Annotating sentences within scientific papers is labor-intensive, resulting in small-scale datasets that limit the amount of information models can learn. This limited information leads models to rely heavily on specific forms, which in turn reduces their generalization capabilities. This paper addresses the problems caused by small-scale datasets from three perspectives: increasing dataset scale, reducing dependence on specific forms, and enriching the information within sentences. To implement the first two ideas, we introduce the concept of formulaic expression (FE) desensitization and propose FE desensitization-based data augmenters to generate synthetic data and reduce models' reliance on FEs. For the third idea, we propose a context-enhanced transformer that utilizes context to measure the importance of words in target sentences and to reduce noise in the context. Furthermore, this paper conducts experiments using large language model (LLM) based in-context learning (ICL) methods. Quantitative and qualitative experiments demonstrate that our proposed models achieve a higher macro F1 score compared to the baseline models on two scientific paper datasets, with improvements of 3.71% and 2.67%, respectively. The LLM based ICL methods are found to be not suitable for the task of problem and method extraction.

Problem

• scientific paper
• problem extraction
• method extraction
• small-scale dataset
• formulaic expression

Innovation

• formulaic expression desensitization
• context-enhanced transformer
• data augmentation
• scientific sentence extraction
• in-context learning