Elevating Legal LLM Responses: Harnessing Trainable Logical Structures and Semantic Knowledge with Legal Reasoning

2025-02-11

AI Summary

Large language models (LLMs) exhibit insufficient logical specificity and suffer from factual hallucinations in legal question answering. Existing retrieval-augmented generation (RAG) methods rely solely on semantic similarity, neglecting the structured logical reasoning essential for legal inference. To address this, we propose the first logic-semantic joint modeling paradigm, comprising trainable fact-rule chain prediction, Deep Structured Semantic Matching (DSSM), and logic-aware RAG—enabling end-to-end co-modeling of logical structure and semantic knowledge. Our method employs reinforcement learning to optimize logical chain reasoning and integrates a structured context generation mechanism. Evaluated on real-world legal QA benchmarks, our framework significantly improves answer accuracy and credibility. Both automated metrics and human evaluation demonstrate consistent superiority over state-of-the-art approaches. This work establishes a novel, reliable reasoning paradigm for legal-domain LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Description Logics
• Natural Language Processing: Question Answering

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) have achieved impressive results across numerous domains, yet they experience notable deficiencies in legal question-answering tasks. LLMs often generate generalized responses that lack the logical specificity required for expert legal advice and are prone to hallucination, providing answers that appear correct but are unreliable. Retrieval-Augmented Generation (RAG) techniques offer partial solutions to address this challenge, but existing approaches typically focus only on semantic similarity, neglecting the logical structure essential to legal reasoning. In this paper, we propose the Logical-Semantic Integration Model (LSIM), a novel supervised framework that bridges semantic and logical coherence. LSIM comprises three components: reinforcement learning predicts a structured fact-rule chain for each question, a trainable Deep Structured Semantic Model (DSSM) retrieves the most relevant candidate questions by integrating semantic and logical features, and in-context learning generates the final answer using the retrieved content. Our experiments on a real-world legal QA dataset-validated through both automated metrics and human evaluation-demonstrate that LSIM significantly enhances accuracy and reliability compared to existing methods.

Problem

• Improves legal question-answering accuracy
• Integrates logical and semantic coherence
• Reduces hallucination in LLM responses

Innovation

• Trainable Logical Structures
• Semantic Knowledge Integration
• Reinforcement Learning Framework