Beyond Accuracy: Measuring Logical Compliance of Predictive Models

2026-06-18

AI Summary

Current model evaluation practices overly emphasize predictive accuracy while neglecting adherence to logical or domain-specific constraints, often leading to unreliable outputs in high-stakes scenarios. To address this limitation, this work proposes the Rule Violation Score (RVS)—a general-purpose evaluation metric that quantifies a model’s compliance with hard and soft logical rules independently of its accuracy. RVS leverages Horn logic to automatically generate SQL queries, making it applicable to knowledge graph link prediction and relational regression tasks, and compatible with rule-based, embedding-based, and neuro-symbolic models alike. Experiments across three benchmarks demonstrate that RVS effectively uncovers differences in logical consistency among models with comparable predictive performance, thereby addressing a critical blind spot in conventional evaluation protocols.

Abstract

Machine learning models are predominantly evaluated through predictive performance metrics such as ranking quality, prediction error, or classification accuracy. While these metrics effectively quantify how closely predictions match the ground truth, they do not assess whether model outputs respect predefined logical or domain-specific constraints. In high-stakes applications, including healthcare, finance, and autonomous systems, logical consistency can be as critical as predictive accuracy, yet no standard metric captures this dimension. We introduce the Rule Violation Score (RVS), a complementary evaluation metric that quantifies the extent to which a predictive model respects a given set of logical rules, independently of predictive accuracy. RVS treats hard rules (strict constraints) and soft rules (statistical regularities) differently, can be evaluated on any dataset and on any predictive model expressed over a relational vocabulary, and can be computed using SQL queries that are automatically generated for Horn rules. Beyond evaluating models, RVS can also evaluate the logical consistency of training datasets and help identify poorly defined rules. We evaluate RVS on three benchmarks covering knowledge graph link prediction and relational regression, including rule-based, embedding-based, and neuro-symbolic predictive models. Our results demonstrate that two models achieving comparable predictive accuracy can exhibit substantially different levels of logical compliance, revealing differences in model behavior that standard metrics fail to capture.

Problem

• logical compliance
• predictive models
• evaluation metrics
• rule violation
• domain constraints

Innovation

• Rule Violation Score
• logical compliance
• predictive model evaluation
• Horn rules
• relational vocabulary