Learning from Negative Samples in Generative Biomedical Entity Linking

2024-08-29

arXiv.org

AI Summary

Generative biomedical entity linking (BioEL) models are typically trained only on positive samples, leading to insufficient discrimination against semantically similar but semantically distinct negative instances. To address this, we propose ANGEL—a novel framework that pioneers the integration of hard negative sampling with generative BioEL. ANGEL constructs high-quality negatives from knowledge bases, incorporates preference optimization (e.g., DPO) for contrastive learning between positive and negative predictions, and supports both pretraining and fine-tuning enhancements. The method synergizes generative modeling (via Transformer-based entity name generation), Top-k candidate sampling, and reward modeling. Evaluated on five standard BioEL benchmarks, ANGEL achieves an average 1.4% improvement in Top-1 accuracy—rising to 1.7% with pretraining enhancement. All code and models are publicly released.

Technology Category

• Natural Language Processing: Generation
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Generative models have become widely used in biomedical entity linking (BioEL) due to their excellent performance and efficient memory usage. However, these models are usually trained only with positive samples, i.e., entities that match the input mention's identifier, and do not explicitly learn from hard negative samples, which are entities that look similar but have different meanings. To address this limitation, we introduce ANGEL (Learning from Negative Samples in Biomedical Generative Entity Linking), the first framework that trains generative BioEL models using negative samples. Specifically, a generative model is initially trained to generate positive entity names from the knowledge base for given input entities. Subsequently, both correct and incorrect outputs are gathered from the model's top-k predictions. Finally, the model is updated to prioritize the correct predictions through preference optimization. Our models outperform the previous best baseline models by up to an average top-1 accuracy of 1.4% on five benchmarks. When incorporating our framework into pre-training, the performance improvement increases further to 1.7%, demonstrating its effectiveness in both the pre-training and fine-tuning stages. The code and model weights are available at https://github.com/dmis-lab/ANGEL.

Problem

• Generative BioEL models lack hard negative sample training
• ANGEL framework incorporates negative samples for improved accuracy
• Preference optimization boosts correct prediction prioritization

Innovation

• Generative BioEL model with negative samples
• Preference optimization for correct predictions
• Top-k predictions for training enhancement