A Neurosymbolic Approach to Natural Language Formalization and Verification

2025-11-12

AI Summary

In highly regulated domains such as finance and healthcare, the inherent stochasticity of large language models (LLMs) impedes their compliant deployment, necessitating formalization of natural language policies and rigorous verification of logical correctness. This paper proposes a two-stage neuro-symbolic framework: first, LLM-driven *runtime automatic formalization*, augmented by human-in-the-loop guidance; second, cross-verification via multiple independent formalizations and semantic equivalence checking to ensure logical consistency. The approach substantially reduces false positives and generates auditable, traceable logical evidence chains. Evaluated on benchmark policy datasets, it achieves over 99% reliability—marking the first demonstration of high-accuracy, traceable, and formally verifiable automated compliance assessment for natural language policies. This work establishes a trustworthy AI pathway for high-stakes operational environments.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large Language Models perform well at natural language interpretation and reasoning, but their inherent stochasticity limits their adoption in regulated industries like finance and healthcare that operate under strict policies. To address this limitation, we present a two-stage neurosymbolic framework that (1) uses LLMs with optional human guidance to formalize natural language policies, allowing fine-grained control of the formalization process, and (2) uses inference-time autoformalization to validate logical correctness of natural language statements against those policies. When correctness is paramount, we perform multiple redundant formalization steps at inference time, cross checking the formalizations for semantic equivalence. Our benchmarks demonstrate that our approach exceeds 99% soundness, indicating a near-zero false positive rate in identifying logical validity. Our approach produces auditable logical artifacts that substantiate the verification outcomes and can be used to improve the original text.

Problem

• Addressing LLM stochasticity for regulated industry policy compliance
• Formalizing natural language policies with human-guided control
• Verifying logical correctness through auditable autoformalization processes

Innovation

• Neurosymbolic framework formalizes policies with human guidance
• Autoformalization validates logical correctness of statements
• Multiple redundant formalizations ensure semantic equivalence checking