Extracting Cause-Effect Pairs from a Sentence with a Dependency-Aware Transformer Model

2025-07-14

AI Summary

Causal phrase extraction is a critical NLP task with broad applications in domains such as law and medicine. Existing supervised approaches inadequately model dependency syntactic structures. To address this, we propose DepBERT—a novel Transformer-based model that explicitly incorporates dependency parse trees into its architecture for the first time. Specifically, DepBERT introduces a dependency-aware attention mechanism to integrate syntactic path information and couples it with a token-level classification layer for end-to-end causal pair identification. This design effectively bridges linguistic priors with deep semantic representations. Extensive experiments on three standard benchmarks demonstrate that DepBERT consistently outperforms state-of-the-art baselines, achieving superior accuracy and F1 scores. The results empirically validate the substantial performance gains conferred by dependency-structure guidance in causal extraction.

Technology Category

• Natural Language Processing: Information Extraction
• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Extracting cause and effect phrases from a sentence is an important NLP task, with numerous applications in various domains, including legal, medical, education, and scientific research. There are many unsupervised and supervised methods proposed for solving this task. Among these, unsupervised methods utilize various linguistic tools, including syntactic patterns, dependency tree, dependency relations, etc. among different sentential units for extracting the cause and effect phrases. On the other hand, the contemporary supervised methods use various deep learning based mask language models equipped with a token classification layer for extracting cause and effect phrases. Linguistic tools, specifically, dependency tree, which organizes a sentence into different semantic units have been shown to be very effective for extracting semantic pairs from a sentence, but existing supervised methods do not have any provision for utilizing such tools within their model framework. In this work, we propose DepBERT, which extends a transformer-based model by incorporating dependency tree of a sentence within the model framework. Extensive experiments over three datasets show that DepBERT is better than various state-of-the art supervised causality extraction methods.

Problem

• Extracting cause-effect pairs from sentences using NLP
• Incorporating dependency trees into transformer models for causality extraction
• Improving accuracy over existing supervised causality extraction methods

Innovation

• Dependency-aware transformer model for causality extraction
• Incorporates dependency tree within transformer framework
• Outperforms state-of-the-art supervised methods