Interaction Concordance Index: Performance Evaluation for Interaction Prediction Methods

2025-10-16

AI Summary

This work addresses the challenge in drug–target affinity (DTA) prediction that existing models fail to capture the directional nature of drug–target interactions. To this end, we propose the Interaction Consistency Index (IC-index), the first metric specifically designed to evaluate a model’s capacity to encode interaction directionality. The IC-index quantifies the consistency of predicted affinity changes with respect to perturbations in drug or target inputs, exposing the invariance flaws of conventional methods under null interactions and highlighting intrinsic limitations of permutation-equivariant architectures for interaction modeling. We systematically validate the IC-index across multiple benchmark datasets using diverse side information—including molecular graphs and protein sequences—and state-of-the-art machine learning models. Experiments demonstrate that the IC-index effectively discriminates models based on their ability to learn interaction structure, complementing traditional metrics (e.g., RMSE, CI) that lack directional sensitivity. This provides a novel paradigm for DTA model development and clinically informed drug–target matching decisions.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Consider two sets of entities and their members' mutual affinity values, say drug-target affinities (DTA). Drugs and targets are said to interact in their effects on DTAs if drug's effect on it depends on the target. Presence of interaction implies that assigning a drug to a target and another drug to another target does not provide the same aggregate DTA as the reversed assignment would provide. Accordingly, correctly capturing interactions enables better decision-making, for example, in allocation of limited numbers of drug doses to their best matching targets. Learning to predict DTAs is popularly done from either solely from known DTAs or together with side information on the entities, such as chemical structures of drugs and targets. In this paper, we introduce interaction directions' prediction performance estimator we call interaction concordance index (IC-index), for both fixed predictors and machine learning algorithms aimed for inferring them. IC-index complements the popularly used DTA prediction performance estimators by evaluating the ratio of correctly predicted directions of interaction effects in data. First, we show the invariance of IC-index on predictors unable to capture interactions. Secondly, we show that learning algorithm's permutation equivariance regarding drug and target identities implies its inability to capture interactions when either drug, target or both are unseen during training. In practical applications, this equivariance is remedied via incorporation of appropriate side information on drugs and targets. We make a comprehensive empirical evaluation over several biomedical interaction data sets with various state-of-the-art machine learning algorithms. The experiments demonstrate how different types of affinity strength prediction methods perform in terms of IC-index complementing existing prediction performance estimators.

Problem

• Evaluates interaction prediction methods for drug-target affinity data
• Assesses accuracy of predicting interaction directions between entities
• Complements existing performance measures with interaction concordance index

Innovation

• Introduced interaction concordance index for evaluation
• Assessed interaction direction prediction in machine learning
• Incorporated side information to remedy permutation equivariance