AIMS.au: A Dataset for the Analysis of Modern Slavery Countermeasures in Corporate Statements

2025-02-10

AI Summary

This study addresses the challenge of automatically identifying substantive anti-slavery measures in modern slavery statements issued by large corporations, distinguishing vague commitments from verifiable actions. Methodologically, it introduces a novel classification task aligned with Australia’s mandatory reporting requirements under the Modern Slavery Act, combining zero-shot prompting and supervised fine-tuning of BERT and LLaMA models. Its key contribution is AIMS.au—the first fine-grained, sentence-level annotated corpus for this domain, comprising 5,731 Australian corporate statements and systematically labeled verifiable anti-slavery measures. The annotation framework explicitly balances legal compliance with model evaluability, offering the first standardized operationalization of “verifiability” in anti-slavery disclosures. On the measure identification task, fine-tuned models achieve an F1 score of 82.3%, outperforming zero-shot baselines by 29.1 percentage points. The dataset is publicly released, establishing a new benchmark for evaluating automated analysis of corporate modern slavery reporting.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Humans and AI: AI for Accessibility
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Despite over a decade of legislative efforts to address modern slavery in the supply chains of large corporations, the effectiveness of government oversight remains hampered by the challenge of scrutinizing thousands of statements annually. While Large Language Models (LLMs) can be considered a well established solution for the automatic analysis and summarization of documents, recognizing concrete modern slavery countermeasures taken by companies and differentiating those from vague claims remains a challenging task. To help evaluate and fine-tune LLMs for the assessment of corporate statements, we introduce a dataset composed of 5,731 modern slavery statements taken from the Australian Modern Slavery Register and annotated at the sentence level. This paper details the construction steps for the dataset that include the careful design of annotation specifications, the selection and preprocessing of statements, and the creation of high-quality annotation subsets for effective model evaluations. To demonstrate our dataset's utility, we propose a machine learning methodology for the detection of sentences relevant to mandatory reporting requirements set by the Australian Modern Slavery Act. We then follow this methodology to benchmark modern language models under zero-shot and supervised learning settings.

Problem

• Analyze modern slavery countermeasures in corporate statements
• Evaluate and fine-tune Large Language Models for statement assessment
• Detect sentences relevant to Australian Modern Slavery Act requirements

Innovation

• Dataset for modern slavery analysis
• Machine learning for mandatory reporting
• Zero-shot and supervised learning benchmarks