Optimizing Large Language Models for ESG Activity Detection in Financial Texts

2025-02-28

AI Summary

Identifying ESG activities—particularly environmental ones—in financial texts remains challenging due to poor generalization of domain-specific large language models and scarcity of high-quality annotated data. Method: We propose a lightweight, controllable fine-tuning paradigm: (i) constructing ESG-Activities, the first fine-grained benchmark dataset grounded in the EU Taxonomy (1,325 real-world annotated samples); (ii) integrating rule-guided synthetic data generation with Taxonomy-aligned annotation; and (iii) performing supervised fine-tuning and domain adaptation on open-source models (e.g., Llama-7B, Gemma-7B). Results: Fine-tuned Llama-7B achieves over 22% accuracy gain, outperforming closed-source models including GPT-4-turbo. To our knowledge, this is the first work demonstrating that small-parameter open-source models can surpass commercial LLMs on ESG activity detection—enabling auditable, low-cost ESG compliance assessment in sustainable finance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The integration of Environmental, Social, and Governance (ESG) factors into corporate decision-making is a fundamental aspect of sustainable finance. However, ensuring that business practices align with evolving regulatory frameworks remains a persistent challenge. AI-driven solutions for automatically assessing the alignment of sustainability reports and non-financial disclosures with specific ESG activities could greatly support this process. Yet, this task remains complex due to the limitations of general-purpose Large Language Models (LLMs) in domain-specific contexts and the scarcity of structured, high-quality datasets. In this paper, we investigate the ability of current-generation LLMs to identify text related to environmental activities. Furthermore, we demonstrate that their performance can be significantly enhanced through fine-tuning on a combination of original and synthetically generated data. To this end, we introduce ESG-Activities, a benchmark dataset containing 1,325 labelled text segments classified according to the EU ESG taxonomy. Our experimental results show that fine-tuning on ESG-Activities significantly enhances classification accuracy, with open models such as Llama 7B and Gemma 7B outperforming large proprietary solutions in specific configurations. These findings have important implications for financial analysts, policymakers, and AI researchers seeking to enhance ESG transparency and compliance through advanced natural language processing techniques.

Problem

• Enhancing ESG activity detection in financial texts using AI.
• Overcoming limitations of general-purpose LLMs in ESG contexts.
• Improving classification accuracy with fine-tuned models and datasets.

Innovation

• Fine-tuning LLMs on ESG-specific datasets
• Using synthetic data to enhance model performance
• Creating ESG-Activities benchmark for classification