Developing an ESG-Oriented Large Language Model through ESG Practices

2026-03-20

AI Summary

This work addresses the limitations of existing large language models in the ESG domain, which are predominantly confined to discriminative tasks and struggle with generative interactions requiring deep domain knowledge and contextual understanding. Building upon the Qwen-3-4B architecture, this study proposes a parameter-efficient fine-tuning paradigm that integrates Low-Rank Adaptation (LoRA) with Instruction Residual Method (IRM), uniquely embedding ESG principles throughout the entire model adaptation pipeline. The resulting ESG-specialized generative model consistently outperforms the base Qwen-3-4B as well as strong baselines such as Llama-3 and Gemma-3 under both zero-shot and knowledge-augmented settings. Comprehensive evaluations demonstrate its superior performance across multiple dimensions, including generation quality, semantic accuracy, and environmental impact alignment.

Technology Category

• Machine Learning: Efficient ML / Green AI
• Natural Language Processing: Generation
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Environmental, Social, and Governance (ESG) considerations play a central role in contemporary financial decision-making. In parallel, Large Language Model (LLM) applications in this domain have primarily emphasized well-defined discriminative tasks, such as classification or scoring, which have proven effective for structured analysis and benchmarking. However, this prevailing focus offers limited support for more interactive and generative ESG scenarios, where embedded domain knowledge and contextual understanding are essential. In this work, we propose an ESG-oriented adaptation pipeline for LLMs that integrates ESG principles not only as a target domain, but also as guiding constraints throughout training and evaluation. Building on the Qwen-3-4B architecture, we explore parameter-efficient adaptation strategies using Low-Rank Adaptation (LoRA) and the Instruction-Residual Method (IRM) to produce three ESG-specialized models. We evaluate the proposed models on ESG question answering under both zero-shot and knowledge-augmented settings, using a diverse set of generative, semantic, readability, and environmental impact metrics. Our results show that the ESG-adapted models consistently outperform their original counterparts and competitive baselines such as Llama-3 and Gemma-3. Although limitations remain in tool-based knowledge integration, this work establishes a foundation for ESG-oriented language generation and highlights the importance of responsible, domain-aware LLM adaptation.

Problem

• ESG
• Large Language Model
• Generative AI
• Domain Adaptation
• Sustainable Finance

Innovation

• ESG-oriented LLM
• Low-Rank Adaptation (LoRA)
• Instruction-Residual Method (IRM)
• domain-aware adaptation
• generative ESG modeling