The Critical Role of Model Selection in Causal Inference: A Comparative Analysis of Classification Models within the InferBERT Framework for Pharmacovigilance

2026-06-15

AI Summary

Distinguishing causal relationships from spurious associations in adverse event reporting remains a significant challenge in pharmacovigilance. This study systematically evaluates the causal inference performance of XGBoost, ALBERT, BioBERT, and Med-LLaMA within the InferBERT framework on the AILF and TRAM datasets using 5-fold cross-validation repeated 20 times, complemented by Do-calculus, isotonic calibration, and conventional signal detection metrics (PRR, ROR, EBGM). The findings reveal that domain-pretrained models—particularly BioBERT—substantially outperform general-purpose large language models like Med-LLaMA, achieving superior accuracy and greater alignment with traditional signal metrics. Although calibration reduces expected calibration error (ECE), its contribution to causal discovery is limited. These results underscore that integrating domain-specific knowledge is more effective than merely scaling model size, thereby challenging the prevailing paradigm that prioritizes large models in pharmacovigilance applications.

Abstract

Distinguishing causal adverse drug events (ADEs) from spurious correlations remains a central challenge in pharmacovigilance. The InferBERT framework integrates transformer models with Do-calculus, but its success hinges on the underlying classification model. This study evaluates the impact of model choice in InferBERT, assessing whether simpler models suffice, if domain-specific pre-training helps, whether scaling to LLMs improves causal detection, and the effect of post-hoc calibration. We performed a comparative study on two benchmarks: Analgesics-induced Acute Liver Failure (AILF) and Tramadol-related Mortalities (TRAM). Four models were evaluated-XGBoost (baseline), ALBERT (original InferBERT), BioBERT (biomedical transformer), and Med-LLaMA (medical LLM)-using 5-fold cross-validation repeated over 20 runs. We measured accuracy, Expected Calibration Error (ECE) pre- and post-isotonic regression, and Jaccard concordance of causal terms with PRR, ROR, and EBGM; significance was tested with paired t-tests. BioBERT achieved the highest accuracy on both datasets, while Med-LLaMA underperformed despite its size and parameter-efficient fine-tuning. Domain-specific pre-training was decisive. Calibration improved ECE but had mixed effects on accuracy and causal discovery. BioBERT's superiority also yielded the strongest concordance with traditional pharmacovigilance signals. These results show that domain-specific pre-training provides a clear advantage over simpler baselines and larger LLMs. Investing in manageable, domain-aware models is more effective for computational pharmacovigilance than simply scaling model size.

Problem

• causal inference
• adverse drug events
• pharmacovigilance
• model selection
• spurious correlations

Innovation

• domain-specific pre-training
• causal inference
• InferBERT
• pharmacovigilance
• model calibration