Neuro-Symbolic Contrastive Learning for Cross-domain Inference

2025-02-11

Electronic Proceedings in Theoretical Computer Science

AI Summary

Pretrained language models (PLMs) exhibit weak generalization, poor robustness, and insufficient logical reasoning in natural language inference (NLI), while inductive logic programming (ILP) suffers from brittle dependence on precise inputs and struggles with noisy or sparse logical spaces. To address these limitations, we propose a differentiable neural-symbolic contrastive learning framework. Our approach is the first to incorporate logic rule sets and logic program embeddings directly into the contrastive learning objective, enabling smooth optimization over discrete logical space and jointly modeling semantic similarity and logical dissimilarity. The method integrates PLM fine-tuning, neural-symbolic computation, logic program embedding, and ILP. Experiments demonstrate substantial improvements in logical accuracy and few-shot generalization across domain-shifted NLI tasks: logical error rates decrease by 32%, and model robustness against perturbations is significantly enhanced.

Technology Category

• Machine Learning: Neuro-Symbolic Learning
• Natural Language Processing: Safety and Robustness
• Knowledge Representation and Reasoning: Logic Programming

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Pre-trained language models (PLMs) have made significant advances in natural language inference (NLI) tasks, however their sensitivity to textual perturbations and dependence on large datasets indicate an over-reliance on shallow heuristics. In contrast, inductive logic programming (ILP) excels at inferring logical relationships across diverse, sparse and limited datasets, but its discrete nature requires the inputs to be precisely specified, which limits their application. This paper proposes a bridge between the two approaches: neuro-symbolic contrastive learning. This allows for smooth and differentiable optimisation that improves logical accuracy across an otherwise discrete, noisy, and sparse topological space of logical functions. We show that abstract logical relationships can be effectively embedded within a neuro-symbolic paradigm, by representing data as logic programs and sets of logic rules. The embedding space captures highly varied textual information with similar semantic logical relations, but can also separate similar textual relations that have dissimilar logical relations. Experimental results demonstrate that our approach significantly improves the inference capabilities of the models in terms of generalisation and reasoning.

Problem

• Bridges PLMs and ILP methods
• Enhances logical inference accuracy
• Improves generalization and reasoning capabilities

Innovation

• Combines neuro-symbolic learning
• Uses logic programs embedding
• Enhances cross-domain inference