Constraint-aware Learning of Probabilistic Sequential Models for Multi-Label Classification

2025-07-20

AI Summary

To address the challenge of modeling and enforcing logical constraints in multi-label classification with large label sets, this paper proposes a joint distribution framework integrating single-label classifiers with probabilistic sequence models. Methodologically, logical constraints are explicitly encoded into both the training objective—via a constraint-guided loss function—and the inference procedure—through constraint-enforced decoding—while sequence modeling captures high-order label dependencies. The key contribution is the first end-to-end framework that simultaneously incorporates constraint information into both training and inference stages, ensuring logical consistency throughout the pipeline. Experiments on multiple large-scale multi-label benchmarks demonstrate significant improvements: average classification accuracy increases by 2.1%, and constraint satisfaction rates improve by up to 37.5%, all while maintaining efficient inference.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

We investigate multi-label classification involving large sets of labels, where the output labels may be known to satisfy some logical constraints. We look at an architecture in which classifiers for individual labels are fed into an expressive sequential model, which produces a joint distribution. One of the potential advantages for such an expressive model is its ability to modelling correlations, as can arise from constraints. We empirically demonstrate the ability of the architecture both to exploit constraints in training and to enforce constraints at inference time.

Problem

• Handling large sets of labels with logical constraints
• Modeling label correlations via sequential probabilistic models
• Enforcing constraints during both training and inference

Innovation

• Constraint-aware learning for sequential models
• Expressive sequential model for joint distribution
• Exploiting and enforcing logical constraints