Parameter Efficient Fine Tuning Llama 3.1 for Answering Arabic Legal Questions: A Case Study on Jordanian Laws

2025-04-28

2025 1st International Conference on Computational Intelligence Approaches and Applications (ICCIAA)

AI Summary

This study addresses the poor adaptability and high computational cost of large language models in Arabic legal question answering, particularly for jurisdiction-specific contexts such as Jordanian law. To this end, the authors construct the first structured question-answering dataset focused on Jordanian law, comprising 6,000 samples. They further propose an efficient fine-tuning approach that uniquely combines parameter-efficient fine-tuning (PEFT) with 4-bit quantization, leveraging the Llama-3.1-8B model enhanced with LoRA adapters and optimized via the Unsloth framework. Experimental results demonstrate that this method substantially reduces computational resource requirements while significantly improving the model’s reasoning capabilities and answer accuracy in Arabic legal QA tasks. The effectiveness of the approach is corroborated by BLEU and ROUGE evaluation metrics.

Technology Category

• Natural Language Processing: Question Answering
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: AI & Law, Justice, Regulation & Governance

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This study uses Jordanian law as a case study to explore the fine-tuning of the Llama-3.1 large language model for Arabic question-answering. Two versions of the model- Llama-3.1-8B-bnb-4bit and Llama-3.1-8B-Instruct-bnb-4bit-were fine-tuned using parameter-efficient fine-tuning (PEFT) with LoRA adapters and 4-bit quantized models, leveraging the Unsloth framework for accelerated and resource-efficient training. A custom dataset of 6000 legal question-answer pairs was curated from Jordanian laws and formatted into structured prompts. Performance was evaluated using the BLEU and the ROUGE metrics to compare the fine-tuned models to their respective base versions. Results demonstrated improved legal reasoning and accuracy while achieving resource efficiency through quantization and optimized fine-tuning strategies. This work underscores the potential of adapting large language models for Arabic legal domains and highlights effective techniques for fine-tuning domain-specific tasks.

Problem

• Parameter Efficient Fine Tuning
• Arabic Legal Question Answering
• Llama 3.1
• Jordanian Laws
• Large Language Models

Innovation

• Parameter-Efficient Fine-Tuning
• LoRA
• 4-bit Quantization
• Arabic Legal QA
• Unsloth