MARRO: Multi-headed Attention for Rhetorical Role Labeling in Legal Documents

2025-03-08

AI Summary

Legal document rhetorical role identification—such as distinguishing facts, arguments, and judgments—is critical for legal understanding and downstream tasks, yet hindered by three key challenges: modeling long documents, domain-specific terminology, and severe scarcity of labeled data. To address these, we propose MARRO, a novel model family integrating multi-head attention with a multi-task learning framework; it introduces label shift prediction as an auxiliary task to enhance sentence-level semantic representation, and incorporates a legal-domain-adapted sequence labeling module using either CRF or Softmax decoding. Evaluated on two benchmark datasets from the Supreme Courts of India and the United Kingdom, MARRO achieves state-of-the-art performance, substantially outperforming both BiLSTM-CRF and single-task Transformer baselines. Our approach effectively mitigates data sparsity and long-range dependency issues inherent in legal text processing.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Knowledge Representation and Reasoning: Argumentation

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Identification of rhetorical roles like facts, arguments, and final judgments is central to understanding a legal case document and can lend power to other downstream tasks like legal case summarization and judgment prediction. However, there are several challenges to this task. Legal documents are often unstructured and contain a specialized vocabulary, making it hard for conventional transformer models to understand them. Additionally, these documents run into several pages, which makes it difficult for neural models to capture the entire context at once. Lastly, there is a dearth of annotated legal documents to train deep learning models. Previous state-of-the-art approaches for this task have focused on using neural models like BiLSTM-CRF or have explored different embedding techniques to achieve decent results. While such techniques have shown that better embedding can result in improved model performance, not many models have focused on utilizing attention for learning better embeddings in sentences of a document. Additionally, it has been recently shown that advanced techniques like multi-task learning can help the models learn better representations, thereby improving performance. In this paper, we combine these two aspects by proposing a novel family of multi-task learning-based models for rhetorical role labeling, named MARRO, that uses transformer-inspired multi-headed attention. Using label shift as an auxiliary task, we show that models from the MARRO family achieve state-of-the-art results on two labeled datasets for rhetorical role labeling, from the Indian and UK Supreme Courts.

Problem

• Identify rhetorical roles in legal documents for better understanding.
• Overcome challenges of unstructured text and specialized legal vocabulary.
• Improve model performance using multi-headed attention and multi-task learning.

Innovation

• Multi-headed attention for legal document analysis
• Multi-task learning with label shift auxiliary task
• Transformer-inspired models for rhetorical role labeling