Verification Learning: Make Unsupervised Neuro-Symbolic System Feasible

2025-03-17

AI Summary

Existing neural-symbolic (NeSy) learning heavily relies on labeled data; in fully unsupervised settings, it suffers from symbolic information loss, explosion of the solution space, and reasoning shortcuts. This paper introduces Verification Learning (VL), the first framework for fully unsupervised NeSy learning: it replaces label-based supervision with rule-based verification and formulates reasoning as a constraint satisfaction problem. Key contributions include: (i) translating symbolic rules into differentiable, optimizable constraints; (ii) designing Dynamic Composition Sorting (DCS), an algorithm that mitigates rule degeneration via adaptive rule composition and ranking; and (iii) introducing a prior alignment mechanism to suppress shortcut learning. VL achieves significant improvements in performance and efficiency across fully unsupervised tasks—including arithmetic, sorting, matching, and chess reasoning. Theoretically, we characterize the applicability boundary of rule-based supervision replacing labels—e.g., feasible for addition but infeasible for Sudoku—establishing necessary conditions for successful unsupervised NeSy learning.

Technology Category

• Machine Learning: Neuro-Symbolic Learning
• Computer Vision: Visual Reasoning & Symbolic Representations
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

The current Neuro-Symbolic (NeSy) Learning paradigm suffers from an over-reliance on labeled data. If we completely disregard labels, it leads to less symbol information, a larger solution space, and more shortcuts-issues that current Nesy systems cannot resolve. This paper introduces a novel learning paradigm, Verification Learning (VL), which addresses this challenge by transforming the label-based reasoning process in Nesy into a label-free verification process. VL achieves excellent learning results solely by relying on unlabeled data and a function that verifies whether the current predictions conform to the rules. We formalize this problem as a Constraint Optimization Problem (COP) and propose a Dynamic combinatorial Sorting (DCS) algorithm that accelerates the solution by reducing verification attempts, effectively lowering computational costs to the level of a Constraint Satisfaction Problem (CSP). To further enhance performance, we introduce a prior alignment method to address potential shortcuts. Our theoretical analysis points out which tasks in Nesy systems can be completed without labels and explains why rules can replace infinite labels, such as in addition, for some tasks, while for others, like Sudoku, the rules have no effect. We validate the proposed framework through several fully unsupervised tasks including addition, sort, match, and chess, each showing significant performance and efficiency improvements.

Problem

• Reduces reliance on labeled data in Neuro-Symbolic Learning
• Introduces Verification Learning for label-free reasoning
• Solves tasks using unlabeled data and rule verification

Innovation

• Verification Learning replaces labels with verification
• Dynamic combinatorial Sorting reduces computational costs
• Prior alignment method prevents shortcut learning